You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 03, 2025 00:41:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3074016995922,31cyclictest3201991-21ssh23:17:540
3074017995240,8cyclictest3253461-21ssh00:30:011
3074018994837,7cyclictest3192626-21systemd-run23:05:012
3074017994839,5cyclictest0-21swapper/100:32:561
3074016994739,5cyclictest3130913-21ssh21:40:010
3074016994718,23cyclictest3187534-21ssh22:58:000
3074018994638,5cyclictest3087811-21idleruntime-cro20:00:012
3074018994637,6cyclictest3188997-21systemd-run23:00:022
3074016994624,15cyclictest12450irq/32-eth000:12:580
3074017994537,4cyclictest0-21swapper/123:37:581
3074017994537,4cyclictest0-21swapper/123:37:581
3074018994435,4cyclictest0-21swapper/200:23:002
3074018994434,6cyclictest3214538-21cron23:35:002
3074018994428,9cyclictest3224195-21sh23:48:032
3074016994424,14cyclictest12450irq/32-eth000:08:010
3074017994335,5cyclictest3116299-21basename21:20:011
3074017994332,8cyclictest3158710-21mailstats22:18:031
3074017994332,7cyclictest1275914-21cups-browsed23:23:441
3074018994237,2cyclictest0-21swapper/200:40:012
3074018994234,5cyclictest3194662-21needreboot23:07:512
3074017994135,3cyclictest0-21swapper/123:45:011
3074017994133,5cyclictest3232914-21latency_hist00:00:031
3074018994032,5cyclictest3098333-21systemd-run20:35:022
3074016994033,4cyclictest3150802-21cut22:07:480
3074016994033,4cyclictest3150802-21cut22:07:480
3074016994033,3cyclictest0-21swapper/023:40:010
3074016994033,3cyclictest0-21swapper/023:40:010
3074016994030,8cyclictest422-21dbus-daemon23:45:010
3074018993930,5cyclictest3152417-21sshd22:09:212
3074018993930,5cyclictest3152417-21sshd22:09:212
3074018993929,6cyclictest422-21dbus-daemon19:55:012
3074018993920,10cyclictest3233095-21sh00:00:062
3074018993918,13cyclictest3165819-21awk22:27:572
307401899383,30cyclictest3133074-21ssh21:42:562
3074018993832,3cyclictest0-21swapper/221:20:012
3074018993827,8cyclictest3130912-21tr21:40:012
307401799384,28cyclictest3258084-21ssh00:37:571
3074017993832,3cyclictest3080035-21kworker/u8:1-nfsiod20:30:011
3074017993830,5cyclictest3209903-21sed23:28:051
3074017993830,5cyclictest3209903-21sed23:28:051
3074017993830,5cyclictest3199887-21grep23:15:011
3074017993830,5cyclictest3108807-21diskmemload22:37:521
3074017993829,6cyclictest1275914-21cups-browsed22:20:361
3074017993829,4cyclictest0-21swapper/121:42:531
3074016993831,5cyclictest3232852-40gzip00:00:020
3074016993825,10cyclictest3244810-21fschecks_time00:17:470
3074016993818,13cyclictest12450irq/32-eth022:17:560
3074018993729,5cyclictest3143339-21aten_r2power_vo21:57:452
3074018993727,6cyclictest3138186-21latency_hist21:50:022
3074017993731,3cyclictest0-21swapper/120:20:011
3074017993729,5cyclictest3118014-21fschecks_count21:22:461
3074017993729,5cyclictest3108807-21diskmemload23:33:091
3074016993729,5cyclictest0-21swapper/022:38:010
3074016993725,6cyclictest12450irq/32-eth023:47:560
307401899363,28cyclictest3162276-21ssh22:23:032
3074018993629,4cyclictest3174614-21latency_hist22:40:012
3074018993628,5cyclictest3185304-21sh22:55:012
3074018993628,4cyclictest3210917-21systemd-run23:30:022
3074018993628,4cyclictest3210917-21systemd-run23:30:022
3074018993627,6cyclictest421-21avahi-daemon20:53:022
3074018993624,9cyclictest3140561-21mailstats21:53:072
307401899362,28cyclictest3158663-21grep22:18:022
307401899361,29cyclictest3176756-21ssh22:42:592
3074017993629,4cyclictest3130891-21ls21:40:011
3074017993627,6cyclictest453-21systemd-logind22:50:011
3074017993626,7cyclictest3081550-21/usr/sbin/munin19:35:021
307401699363,24cyclictest3248781-21ssh00:23:040
3074016993630,4cyclictest422-21dbus-daemon19:35:010
3074016993627,6cyclictest421-21avahi-daemon22:50:300
3074016993624,9cyclictest3258624-21unixbench_multi00:38:090
3074016993619,14cyclictest3162066-21needreboot22:22:550
307401899357,6cyclictest351ktimers/221:27:562
307401899354,9cyclictest36-21ksoftirqd/221:32:592
3074018993527,5cyclictest3119064-21grep21:23:052
3074018993527,5cyclictest1-21systemd23:45:012
3074018993526,6cyclictest3083596-21fschecks_time19:42:472
3074018993519,6cyclictest3253857-21sh00:30:292
3074018993517,6cyclictest171rcu_preempt23:52:202
3074017993528,4cyclictest3243187-21munin-run00:15:011
3074017993528,4cyclictest3189004-21systemd-run23:00:021
3074017993527,4cyclictest0-21swapper/122:32:541
3074017993527,4cyclictest0-21swapper/122:32:541
3074017993526,7cyclictest698-21cron19:55:011
3074017993526,5cyclictest3177386-21unixbench_multi22:43:241
307401699351,26cyclictest3111124-21ssh21:12:550
3074018993428,4cyclictest351ktimers/200:06:212
3074018993427,3cyclictest0-21swapper/220:20:022
3074018993426,5cyclictest3170944-21munin-plugin-st22:35:002
3074018993426,5cyclictest3170944-21munin-plugin-st22:35:002
3074018993424,6cyclictest0-21swapper/222:48:012
307401799343,24cyclictest3140348-21ssh21:53:011
3074017993426,5cyclictest3092295-21cron20:15:011
3074017993425,5cyclictest3165038-21/usr/sbin/munin22:27:561
3074017993424,6cyclictest474-21in:imuxsock22:01:011
3074016993427,4cyclictest0-21swapper/022:12:460
3074016993426,5cyclictest3137113-21ssh21:48:050
3074016993419,11cyclictest0-21swapper/020:25:010
307401899333,21cyclictest3252142-21ssh00:27:582
3074018993327,3cyclictest0-21swapper/219:40:022
3074018993325,5cyclictest3230990-21if_wlan023:57:502
3074018993325,5cyclictest3156364-21systemd-run22:15:012
3074018993325,5cyclictest3111764-21unixbench_singl21:13:082
3074018993324,5cyclictest1275914-21cups-browsed19:22:472
3074018993324,5cyclictest1275914-21cups-browsed19:22:472
3074018993324,3cyclictest171rcu_preempt22:01:102
3074018993322,3cyclictest171rcu_preempt23:38:292
3074018993322,3cyclictest171rcu_preempt23:38:292
3074018993320,7cyclictest351ktimers/200:18:012
3074018993312,11cyclictest3241550-21cut00:12:512
307401799335,21cyclictest3143917-21rm21:57:561
307401799333,23cyclictest657-21snmpd21:27:561
3074017993327,3cyclictest0-21swapper/100:17:291
3074017993325,5cyclictest421-21avahi-daemon20:08:031
3074017993325,5cyclictest3099798-21cron20:40:011
3074017993325,5cyclictest3083134-21latency_hist19:40:021
3074017993324,5cyclictest0-21swapper/121:12:581
3074017993317,3cyclictest171rcu_preempt00:23:521
307401699333,24cyclictest657-21snmpd19:27:560
3074016993325,5cyclictest421-21avahi-daemon00:01:460
3074016993323,7cyclictest3144311-21timerandwakeup21:58:060
307401699331,26cyclictest3191098-21ssh23:02:560
307401899329,8cyclictest3202087-21ntpq23:17:592
307401899327,8cyclictest36-21ksoftirqd/223:22:532
307401899326,9cyclictest36-21ksoftirqd/219:17:522
307401899326,9cyclictest36-21ksoftirqd/219:17:522
307401899324,19cyclictest437-21mta-sts-daemon19:12:502
3074018993216,3cyclictest171rcu_preempt19:32:592
307401799326,20cyclictest3154951-21ssh22:12:591
3074017993225,4cyclictest959-21mta-sts-daemon19:22:591
3074017993225,4cyclictest959-21mta-sts-daemon19:22:591
3074017993224,5cyclictest3085407-21ntp_kernel_err19:47:521
3074017993224,5cyclictest1-21systemd00:10:021
3074017993224,4cyclictest0-21swapper/122:07:491
3074017993224,4cyclictest0-21swapper/122:07:491
3074017993221,7cyclictest421-21avahi-daemon23:04:401
3074017993220,9cyclictest1275914-21cups-browsed22:50:411
3074016993227,2cyclictest0-21swapper/000:35:010
3074016993226,4cyclictest3089196-21grep20:03:050
3074016993224,5cyclictest3213363-21mailstats23:33:080
3074016993223,7cyclictest3133624-21sed21:43:060
3074016993222,7cyclictest3209838-21timerwakeupswit23:28:050
3074016993222,7cyclictest3209838-21timerwakeupswit23:28:050
307401699322,23cyclictest3180229-21awk22:47:590
3074018993116,3cyclictest171rcu_preempt20:38:072
3074017993125,3cyclictest0-21swapper/123:18:001
3074017993125,3cyclictest0-21swapper/121:10:021
3074017993125,3cyclictest0-21swapper/121:10:021
3074017993122,6cyclictest421-21avahi-daemon23:52:201
3074017993122,6cyclictest1275914-21cups-browsed21:45:081
307401799312,23cyclictest3194848-21ssh23:07:571
3074017993121,8cyclictest3125597-21missed_timers21:32:511
3074016993125,4cyclictest956-21mta-sts-daemon19:12:540
3074016993125,4cyclictest3228040-21wc23:53:070
3074016993125,3cyclictest0-21swapper/023:10:010
3074016993119,9cyclictest0-21swapper/022:34:210
3074016993119,9cyclictest0-21swapper/022:34:210
307401899304,10cyclictest36-21ksoftirqd/221:10:012
307401899304,10cyclictest36-21ksoftirqd/221:10:012
3074018993023,4cyclictest0-21swapper/220:45:012
3074018993022,5cyclictest341rcuc/221:05:022
3074018993021,6cyclictest3089223-21grep20:03:052
3074018993021,5cyclictest3199042-21unixbench_singl23:13:122
3074018993017,2cyclictest171rcu_preempt20:47:592
3074018993015,2cyclictest171rcu_preempt20:22:492
3074017993021,6cyclictest3104302-21munin-plugin-st20:55:011
3074016993024,3cyclictest3139533-21/usr/sbin/munin21:53:080
3074016993024,3cyclictest0-21swapper/020:27:550
3074016993020,6cyclictest12450irq/32-eth021:28:010
307401899295,8cyclictest36-21ksoftirqd/219:47:562
3074018992922,4cyclictest341rcuc/220:15:012
3074018992920,6cyclictest453-21systemd-logind21:00:012
3074017992923,3cyclictest3107316-21idleruntime-cro21:05:011
3074017992922,4cyclictest3236443-21grep00:05:011
3074017992922,3cyclictest0-21swapper/119:27:551
3074017992921,5cyclictest1-21systemd20:05:011
3074016992923,4cyclictest3125292-21fschecks_count21:32:480
307401699292,22cyclictest3252061-21sh00:27:550
3074016992918,9cyclictest3082364-21needreboot19:37:510
307401899285,8cyclictest36-21ksoftirqd/220:27:542
307401899284,7cyclictest351ktimers/219:27:482
3074018992811,12cyclictest351ktimers/220:07:532
3074017992821,4cyclictest3101218-21unixbench_multi20:43:051
3074017992821,3cyclictest0-21swapper/123:46:181
3074017992820,5cyclictest3076527-21ps19:17:571
3074017992820,5cyclictest3076527-21ps19:17:571
3074017992720,4cyclictest3087807-21sendmail20:00:011
3074017992719,5cyclictest3084702-21latency_hist19:45:021
3074017992718,6cyclictest474-21in:imuxsock19:10:011
307401699279,8cyclictest12450irq/32-eth022:03:000
3074016992722,2cyclictest0-21swapper/020:50:010
3074016992721,4cyclictest3093743-21unixbench_singl20:18:050
3074016992721,4cyclictest3085095-21sort19:47:470
3074016992721,3cyclictest955-21mta-sts-daemon21:02:560
3074016992720,5cyclictest3199902-21systemd-run23:15:010
3074016992720,4cyclictest3205855-21proc_pri23:23:020
3074016992720,4cyclictest3092182-21unixbench_multi20:13:080
3074016992719,6cyclictest3104980-21latency20:57:520
3074016992719,6cyclictest3073979-21/usr/sbin/munin19:10:010
3074016992719,5cyclictest0-21swapper/022:42:500
307401699271,21cyclictest3165374-21/usr/sbin/munin22:27:480
307401699270,19cyclictest3114863-21ssh21:17:580
3074017992618,5cyclictest3102740-21head20:48:061
3074016992618,6cyclictest591-21sshd21:22:210
3074016992617,7cyclictest0-21swapper/020:35:010
3074016992518,5cyclictest1275914-21cups-browsed19:15:140
3074016992518,5cyclictest1275914-21cups-browsed19:15:140
3074016992518,4cyclictest0-21swapper/020:40:010
307401699251,18cyclictest1994697-21sendmail-mta20:52:580
3074017992411,5cyclictest0-21swapper/120:32:511
307401699242,16cyclictest3083922-21ntpq19:42:530
307401699241,20cyclictest3080035-21kworker/u8:1+rpciod20:08:450
3074016992411,10cyclictest0-21swapper/021:08:050
3074016992411,10cyclictest0-21swapper/021:08:050
3074017992317,3cyclictest0-21swapper/121:00:011
3074017992314,7cyclictest3094866-21processes20:22:591
307401699232,15cyclictest3087263-21ntp_states19:57:520
3074016992316,5cyclictest3101306-21cron20:45:010
3074018992216,3cyclictest0-21swapper/219:10:012
3074017992216,3cyclictest957-21mta-sts-daemon19:12:301
307401699221,18cyclictest3078019-21ntpq19:22:550
307401699221,18cyclictest3078019-21ntpq19:22:550
307401999202,9cyclictest0-21swapper/300:28:013
307401699207,10cyclictest0-21swapper/019:55:010
307401999182,9cyclictest0-21swapper/322:22:593
307401999174,6cyclictest0-21swapper/321:13:013
307401999173,8cyclictest0-21swapper/321:18:023
307401999173,7cyclictest0-21swapper/319:17:573
307401999173,7cyclictest0-21swapper/319:17:573
307401999172,9cyclictest0-21swapper/322:52:563
307401999171,8cyclictest0-21swapper/323:32:583
307401999171,7cyclictest0-21swapper/322:12:483
307401999163,8cyclictest0-21swapper/300:32:563
307401999163,7cyclictest0-21swapper/321:53:013
307401999162,8cyclictest0-21swapper/321:27:563
307401999162,7cyclictest0-21swapper/320:38:003
307401999161,8cyclictest0-21swapper/300:12:563
307401999161,10cyclictest0-21swapper/323:47:563
307401999154,6cyclictest0-21swapper/322:48:003
307401999153,6cyclictest0-21swapper/323:07:593
307401999153,6cyclictest0-21swapper/300:37:563
307401999152,7cyclictest0-21swapper/322:57:563
307401999152,7cyclictest0-21swapper/321:32:543
307401999152,6cyclictest0-21swapper/322:42:523
307401999152,6cyclictest0-21swapper/322:03:033
307401999151,9cyclictest0-21swapper/321:47:563
307401999151,8cyclictest0-21swapper/322:18:023
307401999151,8cyclictest0-21swapper/300:02:573
307401999151,7cyclictest0-21swapper/323:02:593
307401999151,7cyclictest0-21swapper/322:38:003
307401999151,7cyclictest0-21swapper/320:57:583
307401999151,7cyclictest0-21swapper/320:52:593
307401999151,6cyclictest0-21swapper/319:42:543
307401999144,6cyclictest0-21swapper/321:22:553
307401999143,7cyclictest0-21swapper/323:57:563
307401999143,6cyclictest0-21swapper/323:22:553
307401999143,6cyclictest0-21swapper/322:32:553
307401999143,6cyclictest0-21swapper/322:32:553
307401999142,7cyclictest0-21swapper/323:52:563
307401999142,7cyclictest0-21swapper/323:37:553
307401999142,7cyclictest0-21swapper/323:37:553
307401999142,7cyclictest0-21swapper/321:37:573
307401999142,7cyclictest0-21swapper/320:07:573
307401999141,7cyclictest0-21swapper/319:57:493
307401999141,6cyclictest0-21swapper/320:22:513
307401999141,6cyclictest0-21swapper/320:17:543
307401999141,6cyclictest0-21swapper/320:13:013
307401999133,5cyclictest0-21swapper/319:37:583
307401999132,7cyclictest0-21swapper/323:42:573
307401999132,7cyclictest0-21swapper/323:17:553
307401999132,7cyclictest0-21swapper/300:07:563
307401999132,6cyclictest0-21swapper/323:12:223
307401999132,6cyclictest0-21swapper/322:07:553
307401999132,6cyclictest0-21swapper/322:07:553
307401999132,6cyclictest0-21swapper/321:57:483
307401999131,8cyclictest0-21swapper/321:07:543
307401999131,8cyclictest0-21swapper/321:07:543
307401999131,7cyclictest0-21swapper/319:35:013
307401999131,7cyclictest0-21swapper/300:22:593
307401999131,6cyclictest0-21swapper/319:27:553
307401999123,6cyclictest431ktimers/319:10:013
307401999122,6cyclictest0-21swapper/322:27:503
307401999122,6cyclictest0-21swapper/321:05:023
307401999122,6cyclictest0-21swapper/319:47:533
307401999121,7cyclictest0-21swapper/323:27:453
307401999121,7cyclictest0-21swapper/323:27:453
307401999121,6cyclictest0-21swapper/320:42:583
307401999121,6cyclictest0-21swapper/300:17:473
307401999111,6cyclictest0-21swapper/321:42:433
307401999105,3cyclictest0-21swapper/320:02:503
307401999101,6cyclictest0-21swapper/320:27:513
307401999101,6cyclictest0-21swapper/319:12:573
30740199994,3cyclictest0-21swapper/320:45:543
30740199991,5cyclictest0-21swapper/319:22:573
30740199991,5cyclictest0-21swapper/319:22:573
30740199991,3cyclictest0-21swapper/320:33:013
30740199990,7cyclictest0-21swapper/319:55:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional