Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate an error if -R option is provided without the -r option. #237

Open
sandrocom opened this issue Mar 9, 2021 · 4 comments
Open

Generate an error if -R option is provided without the -r option. #237

sandrocom opened this issue Mar 9, 2021 · 4 comments
Labels
good first issue
Milestone
Release 1.7

Comments

@sandrocom
Copy link

We're trying to process a series of .pcap files captured over time and hoped we could use tcpflow for both extracting the payloads and keeping track of the sessions' states at the same time using the written DFXML.

From the man page I understood that using -R on the next file (n) should complete TCP flows. Unfortunately it is not really clear what this exactly means and if this should have effect on previously written flow files for open flows and/or the DFXML.

I expected -R it to look for closing flows (with TCP FIN or RST) and to either append/complete the existing flow files with the remaining payload or writing new ones with %c suffix while ignoring continuing/not closing flow. But maybe I completely missed the point of what -R should do.
Also, whenever I use -R, the DFXML does not contain any flows or references to flow files, and also with -v I don't see any files getting accessed or written, it does not seem to do anything:

vagrant@vagrant:~$ tcpflow -R imap_download-0002.pcap -o output -v
reportfilename: output/report.xml
tcpflow: TCPFLOW version 1.6.1
tcpflow: looking for handler for datalink type 1 for interface imap_download-0002.pcap
tcpflow: process_pkt..............................................................................
/.../
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: Open FDs at end of processing:      0
tcpflow: demux.max_open_flows:               0
tcpflow: Flow map size at end of processing: 0
tcpflow: Flows seen:                         0
tcpflow: Cleaning up flows
tcpflow: Total flows processed: 0
tcpflow: Total packets processed: 0

whereas with -r i get flow files written and a summary like

vagrant@vagrant:~$ tcpflow -r imap_download-0002.pcap -o output -v
reportfilename: output/report.xml
tcpflow: TCPFLOW version 1.6.1
tcpflow: looking for handler for datalink type 1 for interface imap_download-0002.pcap
tcpflow: process_pkt..............................................................................
/.../
tcpflow: Open FDs at end of processing:      0
tcpflow: demux.max_open_flows:               4
tcpflow: Flow map size at end of processing: 0
tcpflow: Flows seen:                         6
tcpflow: Cleaning up flows
tcpflow: Total flows processed: 6
tcpflow: Total packets processed: 32

Could you please explain what -R should exactly and if I should see something in the DFXML?
@simsong
Copy link
Owner

simsong commented Mar 9, 2021

I believe that connections found in files read with -R will only be used to terminate TCP sessions, but a SYN or SYN-ACK seen in the stream will not start a connection.

The idea is that you could record a PCAP file every hour, and run tcpflow with -r for hour n and -R for hour n+1. Repeat. That way each TCP connection appears only in a single location.

@simsong simsong closed this as completed Mar 9, 2021
@simsong
Copy link
Owner

simsong commented Mar 9, 2021

Please feel free to improve the documentation and give a pull request!

@sandrocom
Copy link
Author

Thanks for the feedback. We have the same understanding of the documentation then.

But why -R is not processing anything in the example above? When we look at the example pcap's content we see three open sessions continuing and closing. I would expect -R to process them but it does nothing at all.

vagrant@vagrant:~$ tcpdump -nn -r imap_download-0002.pcap
reading from file imap_download-0002.pcap, link-type EN10MB (Ethernet)
12:33:46.917000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 1639903813:1639904161, ack 1885025558, win 14480, options [nop,nop,TS val 4216987276 ecr 299123683], length 348
12:33:46.933000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 1:25, ack 348, win 5229, options [nop,nop,TS val 299123707 ecr 4216987276], length 24
12:33:46.935000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 348:431, ack 25, win 14480, options [nop,nop,TS val 4216987295 ecr 299123707], length 83
12:33:46.951000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 25:50, ack 431, win 5312, options [nop,nop,TS val 299123725 ecr 4216987295], length 25
12:33:46.953000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 431:506, ack 50, win 14480, options [nop,nop,TS val 4216987313 ecr 299123725], length 75
12:33:46.953000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 506:527, ack 50, win 14480, options [nop,nop,TS val 4216987313 ecr 299123725], length 21
12:33:46.979000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 50:235, ack 527, win 5408, options [nop,nop,TS val 299123753 ecr 4216987313], length 185
12:33:46.982000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 527:1500, ack 235, win 15544, options [nop,nop,TS val 4216987342 ecr 299123753], length 973
12:33:47.012000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 235:243, ack 1500, win 6381, options [nop,nop,TS val 299123787 ecr 4216987342], length 8
12:33:47.015000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 1500:1508, ack 243, win 15544, options [nop,nop,TS val 4216987374 ecr 299123787], length 8
12:33:47.984000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 243:249, ack 1508, win 6389, options [nop,nop,TS val 299124758 ecr 4216987374], length 6
12:33:47.987000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 1508:1529, ack 249, win 15544, options [nop,nop,TS val 4216988347 ecr 299124758], length 21
12:33:48.018000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 249:291, ack 1529, win 6410, options [nop,nop,TS val 299124793 ecr 4216988347], length 42
12:33:48.024000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 1529:2301, ack 291, win 15544, options [nop,nop,TS val 4216988383 ecr 299124793], length 772
12:33:48.024000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 2301:2326, ack 291, win 15544, options [nop,nop,TS val 4216988383 ecr 299124793], length 25
12:33:48.088000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 291:300, ack 2326, win 7207, options [nop,nop,TS val 299124862 ecr 4216988383], length 9
12:33:48.090000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 2326:2334, ack 300, win 15544, options [nop,nop,TS val 4216988450 ecr 299124862], length 8
12:34:00.571000 IP 92.106.177.33.57559 > 195.186.123.151.143: Flags [P.], seq 2098427815:2098427840, ack 4164180607, win 5321, options [nop,nop,TS val 299137344 ecr 4216985765], length 25
12:34:00.573000 IP 195.186.123.151.143 > 92.106.177.33.57559: Flags [P.], seq 1:22, ack 25, win 14480, options [nop,nop,TS val 4217000933 ecr 299137344], length 21
12:34:00.587000 IP 92.106.177.33.57559 > 195.186.123.151.143: Flags [F.], seq 25, ack 22, win 5342, options [nop,nop,TS val 299137359 ecr 4217000933], length 0
12:34:00.587000 IP 195.186.123.151.143 > 92.106.177.33.57559: Flags [F.], seq 22, ack 26, win 14480, options [nop,nop,TS val 4217000947 ecr 299137359], length 0
12:34:00.572000 IP 92.106.177.33.57558 > 195.186.123.151.143: Flags [P.], seq 722546331:722546358, ack 1246994586, win 8596, options [nop,nop,TS val 299137346 ecr 4216981665], length 27
12:34:00.575000 IP 195.186.123.151.143 > 92.106.177.33.57558: Flags [P.], seq 1:23, ack 27, win 15544, options [nop,nop,TS val 4217000934 ecr 299137346], length 22
12:34:00.586000 IP 195.186.123.151.143 > 92.106.177.33.57558: Flags [P.], seq 23:89, ack 27, win 15544, options [nop,nop,TS val 4217000946 ecr 299137348], length 66
12:34:00.586000 IP 195.186.123.151.143 > 92.106.177.33.57558: Flags [F.], seq 89, ack 27, win 15544, options [nop,nop,TS val 4217000946 ecr 299137348], length 0
12:34:00.587000 IP 92.106.177.33.57558 > 195.186.123.151.143: Flags [F.], seq 27, ack 23, win 8618, options [nop,nop,TS val 299137360 ecr 4217000934], length 0
12:34:00.570000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 300:316, ack 2334, win 7215, options [nop,nop,TS val 299137344 ecr 4216988450], length 16
12:34:00.570000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [P.], seq 316:327, ack 2334, win 7215, options [nop,nop,TS val 299137344 ecr 4216988450], length 11
12:34:00.573000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 2334:2356, ack 327, win 15544, options [nop,nop,TS val 4217000933 ecr 299137344], length 22
12:34:00.583000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [P.], seq 2356:2422, ack 327, win 15544, options [nop,nop,TS val 4217000943 ecr 299137347], length 66
12:34:00.583000 IP 195.186.123.151.143 > 92.106.177.33.57560: Flags [F.], seq 2422, ack 327, win 15544, options [nop,nop,TS val 4217000943 ecr 299137347], length 0
12:34:00.586000 IP 92.106.177.33.57560 > 195.186.123.151.143: Flags [F.], seq 327, ack 2423, win 7303, options [nop,nop,TS val 299137360 ecr 4217000943], length 0

Is this because tcpflow -R for (n).pcap is not aware of these sessions were being open/detected in the earlier pcap file (n-1).pcap with -r? Should this state be read from the DFXML whenever tcpflow is run?

This is what I ran first (n-1):

vagrant@vagrant:~$ tcpflow -r imap_download-0001.pcap -o output
reportfilename: output/report.xml
vagrant@vagrant:~$ ls -al output/
total 40
drwxrwxr-x 2 vagrant vagrant 4096 Mar 10 13:42 .
drwxr-xr-x 6 vagrant vagrant 4096 Mar 10 12:41 ..
-rw-rw-r-- 1 vagrant vagrant   85 Mar  8 12:33 092.106.177.033.57558-195.186.123.151.00143--1469
-rw-rw-r-- 1 vagrant vagrant  192 Mar  8 12:33 092.106.177.033.57559-195.186.123.151.00143--1469
-rw-rw-r-- 1 vagrant vagrant  158 Mar  8 12:33 092.106.177.033.57560-195.186.123.151.00143--1469
-rw-rw-r-- 1 vagrant vagrant 2046 Mar  8 12:33 195.186.123.151.00143-092.106.177.033.57558--1469
-rw-rw-r-- 1 vagrant vagrant  941 Mar  8 12:33 195.186.123.151.00143-092.106.177.033.57559--1469
-rw-rw-r-- 1 vagrant vagrant  501 Mar  8 12:33 195.186.123.151.00143-092.106.177.033.57560--1469
-rw-rw-r-- 1 vagrant vagrant 5745 Mar 10 13:42 report.xml

Then for (n):

vagrant@vagrant:~$ tcpflow -R imap_download-0002.pcap -o output -v
reportfilename: output/report.xml
tcpflow: TCPFLOW version 1.6.1
tcpflow: looking for handler for datalink type 1 for interface imap_download-0002.pcap
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: process_pkt..............................................................................
tcpflow: Open FDs at end of processing:      0
tcpflow: demux.max_open_flows:               0
tcpflow: Flow map size at end of processing: 0
tcpflow: Flows seen:                         0
tcpflow: Cleaning up flows
tcpflow: Total flows processed: 0
tcpflow: Total packets processed: 0

@simsong
Copy link
Owner

simsong commented Mar 10, 2021

Using -R by itself will always result in 0 processed, because the R files only close tcp connections created in the r files.

@simsong simsong reopened this Mar 10, 2021
@simsong simsong changed the title Command line option -R unclear Generate an error if -R option is provided without the -r option. Mar 10, 2021
@simsong simsong self-assigned this Mar 10, 2021
@simsong simsong removed their assignment Mar 10, 2021
@simsong simsong added this to the Release 1.7 milestone Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue
Projects
None yet
Milestone
Release 1.7
Development

No branches or pull requests
2 participants
@simsong @sandrocom