You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-22 - 03:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 22, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28982099863,20cyclictest326377-21sendmail-msp22:20:010
28982099863,20cyclictest326377-21sendmail-msp22:20:010
289820998621,13cyclictest366519-21ssh23:50:540
289820998416,6cyclictest16850irq/134-800000000:05:170
289820998229,46cyclictest0-21swapper/023:15:020
289829997911,3cyclictest131rcu_preempt20:35:022
28982099791,39cyclictest311415-21ssh21:45:320
289834997623,14cyclictest357828-21ssh23:31:013
289820997620,12cyclictest307938-21ssh21:37:470
289829997531,39cyclictest3467602-21systemd-journal00:00:002
289829997435,36cyclictest30-21ksoftirqd/223:20:022
28982099744,65cyclictest344166-21latency_hist23:00:020
289820997417,47cyclictest16850irq/134-800000023:16:030
28982099739,15cyclictest363724-21modprobe23:45:000
289820997329,40cyclictest304698-21sed21:30:020
289820997327,16cyclictest380837-21sshd00:23:300
289820997327,16cyclictest380837-21sshd00:23:300
28982099732,6cyclictest1-21systemd21:20:010
289820997311,15cyclictest385795-21modprobe00:35:000
289820997135,33cyclictest16950irq/224-800000000:18:020
289820997129,38cyclictest319862-21latency_hist22:05:020
289820997115,48cyclictest16850irq/134-800000000:12:590
289820997110,5cyclictest16850irq/134-800000023:55:360
289834996919,14cyclictest382774-21ssh00:27:543
28982599692,15cyclictest343281-21ssh22:58:231
289820996928,6cyclictest331541-21sh22:31:290
289829996823,41cyclictest357443-21ssh23:30:102
28982599682,13cyclictest380608-21ssh00:22:591
28982599682,13cyclictest380608-21ssh00:22:581
28982099685,18cyclictest388017-21munin-run00:40:000
289820996826,13cyclictest334163-21ssh22:37:380
289820996826,13cyclictest334163-21ssh22:37:380
289820996825,12cyclictest346081-21ssh23:04:530
28982099682,29cyclictest339645-21latency_hist22:50:020
28982099681,28cyclictest0-21swapper/021:22:470
28982999679,4cyclictest131rcu_preempt19:55:022
28982999675,56cyclictest315591-21ls21:55:032
28982999672,15cyclictest345899-21ssh23:04:152
28982999672,15cyclictest310388-21ssh21:43:332
289825996727,36cyclictest298418-21sh21:15:071
28982599672,14cyclictest306196-21ssh21:33:301
289825996716,47cyclictest210-21jbd2/mmcblk1p2-21:50:031
28982099676,12cyclictest292352-21modprobe20:00:010
28982099676,12cyclictest292352-21modprobe20:00:000
289820996716,14cyclictest3467602-21systemd-journal23:25:010
28982099671,4cyclictest304957-21ssh21:30:170
289829996632,29cyclictest296051-21diskmemload22:11:212
289829996630,31cyclictest290793-21sshd19:28:252
289825996627,34cyclictest296175-21date21:10:031
28982599662,14cyclictest321742-21ssh22:09:281
289825996621,41cyclictest23-21ksoftirqd/121:47:141
28982599661,14cyclictest387251-21ssh00:38:111
28982599661,14cyclictest351210-21ssh23:16:011
28982599661,13cyclictest346737-21ssh23:05:501
28982599661,13cyclictest346737-21ssh23:05:501
28982599661,13cyclictest303112-21ssh21:25:531
289820996614,42cyclictest16850irq/134-800000022:41:090
289829996524,36cyclictest275-21dbus-daemon21:25:012
289829996522,39cyclictest340068-21sh22:50:432
28982999652,16cyclictest326358-21munin-run22:20:012
28982999652,16cyclictest326358-21munin-run22:20:002
28982599652,15cyclictest326979-21ssh22:21:031
28982599652,14cyclictest318776-21ssh22:02:281
28982599652,13cyclictest355744-21ssh23:26:251
28982599652,12cyclictest374568-21ssh00:09:241
28982599651,15cyclictest349585-21ssh23:12:291
28982599651,14cyclictest368594-21ssh23:55:381
28982599651,14cyclictest341682-21ssh22:54:531
28982599651,14cyclictest334872-21ssh22:39:221
28982599651,14cyclictest334872-21ssh22:39:211
28982599651,13cyclictest372327-21ssh00:04:191
28982599651,13cyclictest297229-21ssh21:12:341
28982099656,13cyclictest294953-21modprobe20:50:010
28982099655,12cyclictest3467602-21systemd-journal23:35:000
28982099654,22cyclictest361497-21sendmail23:40:010
28982099654,19cyclictest296093-21latency_hist21:10:010
289820996526,11cyclictest323218-21rm22:12:430
28983499649,23cyclictest380925-21ssh00:23:483
28983499649,23cyclictest380925-21ssh00:23:483
289829996429,31cyclictest342251-21sh22:55:402
28982999642,19cyclictest306112-21ssh21:33:192
289825996424,35cyclictest275-21dbus-daemon20:25:001
28982599641,8cyclictest375425-21sshd00:11:001
28982599641,14cyclictest359083-21ssh23:34:161
28982599641,14cyclictest345462-21ssh23:03:161
28982599641,14cyclictest306923-21ssh21:35:091
28982599641,14cyclictest300795-21ssh21:20:341
28982599641,13cyclictest367388-21ssh23:53:011
28982599641,13cyclictest338153-21ssh22:46:291
28982599641,12cyclictest383410-21ssh00:29:361
28982099645,19cyclictest315437-21cut21:55:010
28982099645,15cyclictest293391-21modprobe20:19:590
28982099643,6cyclictest320558-21sshd22:06:250
289820996426,20cyclictest0-21swapper/023:26:460
28982099642,57cyclictest329261-21ssh22:26:190
289820996425,6cyclictest317690-21latency_hist22:00:020
289834996322,14cyclictest364-21snmpd21:47:523
28982999632,13cyclictest363725-21modprobe23:45:002
28982599632,14cyclictest330566-21ssh22:29:381
28982599632,13cyclictest361982-21ssh23:40:351
28982599632,13cyclictest331572-21ssh22:31:341
28982599631,14cyclictest359592-21ssh23:35:081
28982599631,14cyclictest353440-21ssh23:21:001
28982599631,12cyclictest365784-21ssh23:49:331
28982099635,23cyclictest291067-21cut19:35:010
28982099634,19cyclictest293676-21grep20:25:010
28982099632,6cyclictest381749-21ssh00:25:130
289820996320,34cyclictest16850irq/134-800000022:51:550
28982099631,29cyclictest0-21swapper/020:30:000
28982999624,12cyclictest48-21kcompactd000:35:052
289829996226,32cyclictest335075-21sh22:39:502
289829996226,32cyclictest335075-21sh22:39:502
289829996223,34cyclictest3467562-21sshd20:12:242
289829996223,34cyclictest3467562-21sshd20:12:232
28982999621,15cyclictest297805-21ssh21:14:032
28982599622,15cyclictest290572-21munin-run19:25:011
28982599622,13cyclictest336869-21ssh22:43:401
28982599622,13cyclictest295748-21apt-get21:04:261
28982599621,14cyclictest310978-21ssh21:44:571
28982599621,13cyclictest325581-21ssh22:18:061
28982599621,13cyclictest325581-21ssh22:18:051
28982599621,12cyclictest378075-21ssh00:17:101
28982599621,10cyclictest322337-21ssh22:10:201
28982099626,13cyclictest348385-21ssh23:10:020
28982099626,13cyclictest348385-21ssh23:10:020
28982999618,7cyclictest131rcu_preempt22:50:012
28982999615,21cyclictest368168-21munin-plugin-st23:55:002
289829996114,5cyclictest296051-21diskmemload22:00:472
28982999611,14cyclictest326943-21ssh22:20:572
28982599612,12cyclictest383872-21ssh00:30:091
28982599612,12cyclictest316595-21ssh21:57:301
28982999602,18cyclictest379957-21ssh00:21:152
28982999602,18cyclictest379957-21ssh00:21:152
28982999602,16cyclictest373938-21ssh00:07:502
28982099605,16cyclictest294712-21grep20:44:590
289820996031,26cyclictest0-21swapper/021:00:010
289820996031,26cyclictest0-21swapper/021:00:000
289820996024,5cyclictest327376-21sh22:21:550
289820996022,10cyclictest12-21ksoftirqd/000:04:190
289820996019,8cyclictest16950irq/224-800000023:50:010
289820996016,12cyclictest297165-21ssh21:12:250
28982999593,12cyclictest284-21systemd-logind19:20:002
28982999593,10cyclictest348409-21grep23:10:022
28982999593,10cyclictest348409-21grep23:10:022
28982999592,16cyclictest321174-21ssh22:08:002
28982999592,15cyclictest292351-21latency_hist20:00:012
28982999592,15cyclictest292351-21latency_hist20:00:002
289829995919,36cyclictest356829-21sh23:29:032
289829995916,9cyclictest349214-1kworker/u9:123:12:222
28982599592,6cyclictest275-21dbus-daemon19:55:001
28982599592,14cyclictest291859-21munin-run19:50:011
28982599592,10cyclictest294474-21munin-run20:40:021
28982599591,12cyclictest364-21snmpd20:34:561
28982099599,17cyclictest16850irq/134-800000021:40:400
28982099597,14cyclictest289824-21latency_hist19:10:010
28982099591,3cyclictest0-21swapper/019:40:000
289829995823,30cyclictest3467602-21systemd-journal21:26:022
28982999582,15cyclictest377622-21modprobe00:16:012
28982999582,15cyclictest372483-21ssh00:04:392
28982999582,11cyclictest361496-21latency_hist23:40:012
289829995816,4cyclictest375940-21sh00:12:212
289829995816,36cyclictest0-21swapper/220:20:002
28982599581,13cyclictest294964-21/usr/sbin/munin20:50:001
28982099586,11cyclictest1-21systemd19:20:010
289820995851,4cyclictest16950irq/224-800000019:30:010
289834995738,15cyclictest332815-21ssh22:34:463
28982999574,9cyclictest296092-21grep21:10:022
28982999573,12cyclictest284-21systemd-logind19:15:012
289829995728,26cyclictest30-21ksoftirqd/223:22:552
289829995723,30cyclictest30-21ksoftirqd/200:29:592
28982999572,13cyclictest295468-21latency_hist21:00:012
28982999572,13cyclictest295468-21latency_hist21:00:012
289829995718,10cyclictest30-21ksoftirqd/223:45:522
289829995712,13cyclictest3467602-21systemd-journal19:35:012
28982999571,17cyclictest308523-21ssh21:39:182
289825995730,24cyclictest289885-21latency_hist19:10:021
28982599572,15cyclictest293923-21latency_hist20:30:021
28982599572,13cyclictest364-21snmpd19:56:281
28982599572,13cyclictest364-21snmpd19:56:271
28982599572,13cyclictest295473-21sendmail21:00:011
28982599572,13cyclictest295473-21sendmail21:00:001
289825995714,37cyclictest290284-21run-parts19:20:011
28982099574,15cyclictest1-21systemd19:15:010
28982099573,50cyclictest0-21swapper/020:40:010
289834995644,9cyclictest307292-21ssh21:35:593
289834995643,10cyclictest327516-21ssh22:22:263
289834995635,17cyclictest378256-21ssh00:17:353
28982999563,16cyclictest385797-21grep00:35:002
28982999563,13cyclictest294714-21munin-plugin-st20:44:592
289829995619,9cyclictest291761-21sed19:45:032
289829995619,9cyclictest291761-21sed19:45:022
289829995617,4cyclictest296051-21diskmemload22:26:132
289829995615,9cyclictest332221-21sh22:33:112
289829995614,5cyclictest295084-21latency_hist20:50:022
289829995614,12cyclictest313358-21cat21:50:022
28982999561,14cyclictest337380-21/usr/sbin/munin22:44:592
28982999561,14cyclictest1-21systemd20:30:002
28982599563,7cyclictest364-21snmpd19:32:511
28982599561,13cyclictest364-21snmpd20:01:581
28982599561,12cyclictest293101-21sshd20:11:241
28982599561,12cyclictest293101-21sshd20:11:231
28982099564,21cyclictest1-21systemd19:50:010
289820995618,4cyclictest292627-21latency_hist20:05:010
289834995535,16cyclictest351117-21ssh23:15:483
289834995535,15cyclictest314469-21ssh21:52:383
28982999553,15cyclictest300326-21munin-plugin-st21:19:592
289829995515,5cyclictest294420-21cron20:40:002
28982599553,14cyclictest295216-21munin-run20:55:001
28982599552,9cyclictest364-21snmpd19:35:461
289825995513,36cyclictest309-21in:imuxsock19:28:241
28982099554,15cyclictest295237-21wc20:55:010
289834995442,9cyclictest321792-21ssh22:09:363
289834995437,13cyclictest359839-21ssh23:35:413
289834995434,16cyclictest329692-21ssh22:27:253
289834995433,17cyclictest371092-21ssh00:01:103
28982999543,10cyclictest295858-21grep21:05:002
28982999542,21cyclictest292637-21true20:05:002
28982999542,13cyclictest291317-21latency_hist19:40:002
28982999542,11cyclictest317803-21latency_hist22:00:012
289834995337,13cyclictest319236-21ssh22:03:303
289834995336,13cyclictest343883-21ssh22:59:543
289834995335,14cyclictest340285-21ssh22:51:223
289834995335,14cyclictest336078-21ssh22:41:433
289834995334,15cyclictest353136-21ssh23:20:183
289834995334,15cyclictest350494-21ssh23:14:553
289834995333,16cyclictest296524-21ssh21:10:323
289834995332,17cyclictest375556-21ssh00:11:183
289834995332,16cyclictest315962-21ssh21:55:553
28982999533,19cyclictest291859-21unin-run19:50:012
28982599532,11cyclictest290002-21cron19:14:591
28982599531,12cyclictest291581-21cron19:45:011
28982599531,12cyclictest291581-21cron19:45:001
28982099537,15cyclictest3467602-21systemd-journal20:35:010
289834995239,10cyclictest48-21kcompactd023:01:003
289834995238,11cyclictest298463-21ssh21:15:143
289834995237,12cyclictest346413-21ssh23:05:083
289834995237,12cyclictest346413-21ssh23:05:083
289834995236,13cyclictest301762-21ssh21:22:593
289834995236,12cyclictest322386-21ssh22:10:363
289834995235,14cyclictest309053-21ssh21:40:063
289834995235,13cyclictest325243-21ssh22:17:203
289834995235,13cyclictest325243-21ssh22:17:203
289834995235,13cyclictest306271-21ssh21:33:413
289834995233,15cyclictest362844-21ssh23:42:383
289834995232,16cyclictest369581-21ssh23:58:013
289834995231,17cyclictest384351-21ssh00:31:273
289834995231,17cyclictest367216-21ssh23:52:383
289829995220,28cyclictest290567-21systemd-run19:25:002
28982599521,13cyclictest294678-21sshd20:40:251
28982599521,11cyclictest293370-21sshd20:17:251
289834995138,10cyclictest356150-21ssh23:27:193
289834995136,12cyclictest386422-21ssh00:36:093
289834995135,13cyclictest372999-21ssh00:05:283
289834995135,13cyclictest364464-21ssh23:46:123
289834995132,15cyclictest1-21systemd20:50:003
289834995037,10cyclictest294403-21sshd20:36:413
289834995036,11cyclictest302884-21ssh21:25:213
289834995036,10cyclictest291565-21sshd19:41:303
289834995036,10cyclictest291565-21sshd19:41:303
289834995033,13cyclictest338769-21ssh22:48:023
289834995032,15cyclictest334753-21ssh22:38:573
289834995032,15cyclictest334753-21ssh22:38:563
28982999509,6cyclictest295203-21cron20:54:592
289834994936,10cyclictest295446-21sshd20:58:243
289834994936,10cyclictest295446-21sshd20:58:233
289834994936,10cyclictest293351-21sshd20:16:243
289834994935,10cyclictest293905-21sshd20:28:243
28982999493,15cyclictest293675-21munin-plugin-st20:25:012
28982999493,11cyclictest293087-21latency_hist20:10:032
28982099493,14cyclictest295798-21munin-run21:05:000
28982099491,30cyclictest0-21swapper/019:21:260
289834994836,9cyclictest290530-21sshd19:21:273
289834994835,9cyclictest48-21kcompactd019:50:143
289834994835,10cyclictest292847-21sshd20:05:253
28982099481,31cyclictest0-21swapper/019:52:400
289834994734,9cyclictest48-21kcompactd020:14:393
289834994734,9cyclictest48-21kcompactd020:14:383
289834994734,9cyclictest48-21kcompactd019:38:353
289834994729,15cyclictest364-21snmpd19:12:193
289834994725,15cyclictest37-21ksoftirqd/319:30:013
28982599478,3cyclictest0-21swapper/120:05:521
289834994633,9cyclictest48-21kcompactd019:31:223
289834994633,9cyclictest364-21snmpd20:04:193
28982099461,4cyclictest0-21swapper/019:45:020
28982099461,4cyclictest0-21swapper/019:45:020
289834994538,4cyclictest294711-21munin-run20:45:003
289834994535,6cyclictest290319-21munin-run19:20:023
289834994534,8cyclictest48-21kcompactd021:05:403
289834994533,9cyclictest364-21snmpd20:20:553
289834994523,17cyclictest295386-21date20:55:023
289834994437,4cyclictest292357-21munin-run20:00:013
289834994437,4cyclictest292357-21munin-run20:00:003
289834994429,12cyclictest364-21snmpd20:30:563
289834994329,10cyclictest37-21ksoftirqd/321:05:013
289820994320,20cyclictest12-21ksoftirqd/020:10:010
289834994229,9cyclictest364-21snmpd19:47:483
289820994225,4cyclictest0-21swapper/020:13:060
289820994225,4cyclictest0-21swapper/020:13:060
289834993317,13cyclictest289891-21cat19:10:023
289829993021,6cyclictest289879-21latency_hist19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional