You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 15:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Sep 05, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5324799910654,41cyclictest12-21ksoftirqd/010:30:030
53248199894,21cyclictest131rcu_preempt11:50:031
532489998818,36cyclictest559710-21latency_hist09:55:023
532479998528,37cyclictest556575-21ssh09:48:200
532479998528,37cyclictest556575-21ssh09:48:190
532486998372,5cyclictest30-21ksoftirqd/211:30:022
53247999832,9cyclictest635492-21munin-run12:40:020
532479998218,11cyclictest3467602-21systemd-journal12:13:060
532479998026,50cyclictest594483-21ssh11:10:490
53248999792,43cyclictest0-21swapper/310:03:263
53248999791,47cyclictest0-21swapper/310:25:143
532479997932,42cyclictest275-21dbus-daemon08:40:010
532479997932,42cyclictest275-21dbus-daemon08:40:000
532479997918,6cyclictest16850irq/134-800000012:33:490
532479997916,6cyclictest16850irq/134-800000010:16:410
532479997827,33cyclictest551502-21ssh09:37:150
53248999771,45cyclictest632856-21sh12:34:153
53247999762,69cyclictest626051-21ssh12:19:390
532479997626,34cyclictest542641-21ssh09:18:290
532479997626,32cyclictest572135-21ssh10:22:080
532479997626,32cyclictest572135-21ssh10:22:070
532479997622,12cyclictest628900-21ssh12:25:310
53248999751,41cyclictest549986-21ssh09:34:063
53247999753,8cyclictest275-21dbus-daemon10:55:000
532479997526,43cyclictest607828-21cron11:40:010
532479997526,43cyclictest607828-21cron11:40:000
532479997525,32cyclictest627589-21ssh12:22:470
53248999731,47cyclictest577838-21ssh10:34:533
532481997336,27cyclictest23-21ksoftirqd/112:10:021
532479997326,42cyclictest580169-21ssh10:39:550
532479997228,40cyclictest588550-21ssh10:57:480
532479997228,40cyclictest567198-21ssh10:11:180
532479997225,30cyclictest598132-21ssh11:18:570
532479997223,17cyclictest549522-21ssh09:33:080
53247999721,54cyclictest590006-21ssh11:00:430
53247999721,53cyclictest0-21swapper/011:26:170
53248699714,42cyclictest30-21ksoftirqd/207:10:012
532479997125,28cyclictest554827-21ssh09:44:360
532479997124,2cyclictest0-21swapper/011:51:130
532479997123,15cyclictest584478-21ssh10:49:080
53248999701,43cyclictest364-21snmpd10:46:133
53248999701,39cyclictest606574-21ssh11:37:003
53248999701,39cyclictest606574-21ssh11:36:593
53247999702,7cyclictest536054-21munin-run08:20:000
532479997026,28cyclictest559418-21ssh09:54:340
532479997015,37cyclictest16850irq/134-800000011:05:110
53248999692,41cyclictest628088-21ssh12:24:003
53248999691,42cyclictest610179-21ssh11:44:583
53247999691,64cyclictest0-21swapper/011:55:030
53247999691,62cyclictest0-21swapper/010:09:300
532489996828,36cyclictest571753-21sshd10:21:063
532489996828,36cyclictest571753-21sshd10:21:053
532479996837,27cyclictest16950irq/224-800000008:10:050
532479996837,27cyclictest16950irq/224-800000008:10:050
532479996826,38cyclictest561710-21sh09:59:460
532479996826,38cyclictest561710-21sh09:59:450
532479996816,14cyclictest609478-21ssh11:43:200
53247999672,5cyclictest619382-21munin-run12:05:010
53248999666,26cyclictest131rcu_preempt10:00:013
53248999666,26cyclictest131rcu_preempt10:00:003
53248999662,58cyclictest596980-21sshd11:16:063
53248999661,60cyclictest629511-21ssh12:26:593
53247999662,59cyclictest562174-21ssh10:00:120
532479996623,6cyclictest0-21swapper/009:11:210
532479996617,13cyclictest546441-21ssh09:26:160
53248999653,57cyclictest564726-21sshd10:05:533
53248999651,60cyclictest591116-21ssh11:03:323
53248999651,58cyclictest613592-21ssh11:52:073
53248999651,35cyclictest587449-21ssh10:55:183
532479996518,13cyclictest580694-21ssh10:40:290
53248999641,59cyclictest620429-21ssh12:07:003
53248999641,59cyclictest604213-21ssh11:31:533
53248999641,59cyclictest600892-21ssh11:24:523
53248999641,59cyclictest586057-21ssh10:52:253
53248999641,59cyclictest568729-21ssh10:14:583
53248999641,59cyclictest543625-21ssh09:20:123
53248999641,59cyclictest537065-21sshd08:39:053
53248999641,59cyclictest537065-21sshd08:39:053
53248999641,58cyclictest635110-21ssh12:39:073
53248999641,58cyclictest616261-21ssh11:58:053
53248999641,58cyclictest592959-21ssh11:07:363
53248999641,58cyclictest541755-21ssh09:16:193
53248999641,57cyclictest617361-21sshd12:00:063
53247999644,20cyclictest533954-21modprobe07:40:000
532479996427,33cyclictest537966-21cat08:55:020
532479996427,33cyclictest537966-21cat08:55:020
532479996419,11cyclictest577485-21ssh10:33:570
53248999633,56cyclictest602469-21sshd11:28:053
53248999631,59cyclictest569041-21ssh10:15:083
53248999631,58cyclictest611351-21ssh11:47:103
53248999631,58cyclictest595994-21ssh11:14:183
53248999631,58cyclictest555316-21ssh09:45:243
53248999631,58cyclictest555316-21ssh09:45:243
53248999631,58cyclictest553762-21ssh09:42:023
53247999633,22cyclictest0-21swapper/008:25:030
53247999633,22cyclictest0-21swapper/008:25:020
532479996318,10cyclictest538117-21cron08:59:590
53248999621,57cyclictest624873-21ssh12:16:583
53248999621,57cyclictest623881-21ssh12:14:543
53248999621,57cyclictest582123-21ssh10:44:033
53248999621,57cyclictest578415-21ssh10:35:383
53248999621,56cyclictest539850-21ssh09:12:283
53248999621,56cyclictest536293-21sshd08:20:383
53248999621,56cyclictest536293-21sshd08:20:383
53248999621,56cyclictest535502-21sshd08:06:053
53248199625,42cyclictest221rcuc/109:10:021
532481996239,16cyclictest3467602-21systemd-journal10:05:001
532479996225,34cyclictest12-21ksoftirqd/008:45:040
532479996215,42cyclictest16850irq/134-800000009:21:050
53248999612,55cyclictest537881-21munin-run08:55:013
53248999612,55cyclictest537881-21munin-run08:55:013
53248999612,54cyclictest364-21snmpd08:33:003
53248999612,53cyclictest537614-21munin-run08:50:013
53248999611,56cyclictest550756-21ssh09:35:283
53248999611,56cyclictest546109-21ssh09:25:353
53248999611,55cyclictest537361-21munin-run08:45:003
532479996121,4cyclictest611623-21sh11:47:560
532479996119,38cyclictest605432-21rm11:34:360
53248999602,53cyclictest538160-21munin-run09:00:013
53248999602,53cyclictest535293-21munin-run08:05:003
532486996052,5cyclictest30-21ksoftirqd/209:50:032
532486996052,5cyclictest30-21ksoftirqd/209:50:022
53247999602,53cyclictest615142-21kworker/u8:312:09:500
53248999591,35cyclictest364-21snmpd08:11:543
53248999591,35cyclictest364-21snmpd08:11:543
53248999583,50cyclictest364-21snmpd07:47:483
53247999581,20cyclictest538653-21latency_hist09:10:010
53248999572,51cyclictest364-21snmpd07:13:373
53248999572,50cyclictest533725-21munin-run07:35:013
53248999572,50cyclictest364-21snmpd08:17:073
53248999571,52cyclictest364-21snmpd07:37:453
532486995748,6cyclictest548034-21kworker/2:110:31:392
53248699572,8cyclictest30-21ksoftirqd/209:05:012
532481995736,16cyclictest1-21systemd10:35:011
532481995716,14cyclictest131rcu_preempt09:35:021
53247999573,49cyclictest0-21swapper/007:35:010
532479995717,6cyclictest16850irq/134-800000011:24:330
53248999562,49cyclictest538623-21sshd09:07:053
53248999561,50cyclictest364-21snmpd07:15:043
53247999562,23cyclictest533328-21latency_hist07:25:010
53248999558,42cyclictest0-21swapper/308:28:183
532489995537,13cyclictest3467602-21systemd-journal07:45:003
532489995537,13cyclictest3467602-21systemd-journal07:45:003
53248999552,48cyclictest364-21snmpd07:28:173
53248999552,48cyclictest364-21snmpd07:21:233
53248699553,14cyclictest30-21ksoftirqd/211:00:012
53248999542,47cyclictest534849-21ls07:55:013
53247999544,20cyclictest534258-21unin-run07:45:020
53247999544,20cyclictest534258-21unin-run07:45:010
532479995419,4cyclictest533479-21latency_hist07:30:010
53248699538,8cyclictest30-21ksoftirqd/210:45:012
532486995346,5cyclictest532698-21kworker/2:309:54:052
532486995345,5cyclictest532698-21kworker/2:309:29:282
532486995344,6cyclictest580267-21kworker/2:011:24:522
532486995344,5cyclictest532698-21kworker/2:308:49:282
532486995344,5cyclictest295045-21kworker/2:107:50:432
532481995332,15cyclictest275-21dbus-daemon09:05:001
532481995331,16cyclictest571063-21cron10:20:001
532479995315,8cyclictest12-21ksoftirqd/009:05:020
53247999531,4cyclictest0-21swapper/007:48:530
53248999521,46cyclictest364-21snmpd09:02:103
532486995245,4cyclictest548034-21kworker/2:110:54:272
532486995243,6cyclictest532698-21kworker/2:309:34:012
532486995243,6cyclictest30-21ksoftirqd/212:27:072
53248699521,18cyclictest0-21swapper/212:04:002
532479995229,18cyclictest532934-21latency_hist07:20:020
53247999521,21cyclictest535239-21latency_hist08:00:020
532486995144,4cyclictest610219-21kworker/2:112:09:252
532486995143,5cyclictest610219-21kworker/2:112:12:312
532486995143,5cyclictest610219-21kworker/2:111:50:552
532486995143,5cyclictest580267-21kworker/2:011:36:432
532486995143,5cyclictest580267-21kworker/2:011:36:432
532486995143,5cyclictest548034-21kworker/2:110:09:362
53248699512,12cyclictest30-21ksoftirqd/210:20:002
532479995138,10cyclictest12-21ksoftirqd/007:15:010
532486995043,5cyclictest594055-21kworker/2:211:19:112
532486995043,5cyclictest532698-21kworker/2:310:04:082
532486995042,5cyclictest610219-21kworker/2:111:56:142
532486995042,5cyclictest580267-21kworker/2:011:41:522
532486995042,5cyclictest580267-21kworker/2:011:10:052
532486995042,5cyclictest548034-21kworker/2:111:06:492
532486995042,5cyclictest548034-21kworker/2:110:46:482
532486995042,5cyclictest532698-21kworker/2:310:14:112
532486995042,5cyclictest532698-21kworker/2:309:59:532
532486995042,5cyclictest532698-21kworker/2:309:59:522
532486995042,5cyclictest532698-21kworker/2:307:41:212
532486995042,5cyclictest532698-21kworker/2:307:41:212
532486995041,5cyclictest580267-21kworker/2:012:32:432
53248699501,10cyclictest30-21ksoftirqd/208:59:592
53248199504,34cyclictest635593-21ssh12:40:021
532481995036,10cyclictest1-21systemd11:45:001
532479995018,6cyclictest12-21ksoftirqd/008:45:020
532479995017,5cyclictest16850irq/134-800000008:25:050
53248999492,44cyclictest364-21snmpd07:57:493
53248999492,44cyclictest364-21snmpd07:57:493
532486994942,4cyclictest548034-21kworker/2:110:20:412
532486994942,4cyclictest548034-21kworker/2:110:20:412
532486994942,4cyclictest536088-21kworker/2:209:15:292
532486994941,5cyclictest610219-21kworker/2:112:37:312
532486994941,5cyclictest610219-21kworker/2:112:18:512
532486994941,5cyclictest580267-21kworker/2:012:22:162
532486994941,5cyclictest580267-21kworker/2:011:47:262
532486994941,5cyclictest580267-21kworker/2:011:34:022
532486994941,5cyclictest580267-21kworker/2:011:01:522
532486994941,5cyclictest548034-21kworker/2:110:27:082
532486994941,5cyclictest536088-21kworker/2:209:06:542
532486994941,5cyclictest532698-21kworker/2:309:10:232
532486994941,5cyclictest532698-21kworker/2:308:20:022
532486994941,5cyclictest532698-21kworker/2:308:20:022
532486994941,5cyclictest532698-21kworker/2:308:00:062
532486994940,6cyclictest536088-21kworker/2:209:24:332
532486994940,6cyclictest532698-21kworker/2:307:29:192
53248699491,23cyclictest0-21swapper/209:35:482
53247999493,42cyclictest3467602-21systemd-journal08:10:000
532479994919,8cyclictest12-21ksoftirqd/008:35:020
532486994840,5cyclictest548034-21kworker/2:110:37:102
532486994840,5cyclictest548034-21kworker/2:109:40:192
532486994840,5cyclictest532698-21kworker/2:308:35:022
532486994840,5cyclictest532698-21kworker/2:308:35:022
532486994840,5cyclictest532698-21kworker/2:308:34:352
532486994840,5cyclictest532698-21kworker/2:308:10:072
532486994840,5cyclictest532698-21kworker/2:308:10:062
532486994840,5cyclictest532698-21kworker/2:308:07:032
532486994840,5cyclictest532698-21kworker/2:307:30:472
532486994837,5cyclictest295045-21kworker/2:107:18:202
532481994833,11cyclictest3467602-21systemd-journal08:15:051
532481994832,11cyclictest1-21systemd07:15:011
532481994830,13cyclictest1-21systemd11:35:001
53248999472,41cyclictest532606-21latency_hist07:10:023
532486994739,6cyclictest532698-21kworker/2:307:55:372
532486994739,6cyclictest532698-21kworker/2:307:55:372
532486994739,5cyclictest532698-21kworker/2:308:50:042
532486994739,5cyclictest532698-21kworker/2:308:50:032
532486994739,5cyclictest532698-21kworker/2:308:41:472
532486994739,5cyclictest532698-21kworker/2:308:25:392
532486994739,5cyclictest532698-21kworker/2:308:17:032
532486994739,5cyclictest295045-21kworker/2:107:48:092
532486994739,5cyclictest295045-21kworker/2:107:35:272
532486994739,5cyclictest295045-21kworker/2:107:11:292
532486994638,5cyclictest295045-21kworker/2:107:20:262
532481994631,10cyclictest566494-21cat10:10:021
532481994530,11cyclictest23-21ksoftirqd/110:46:161
532481994530,11cyclictest23-21ksoftirqd/109:48:421
532481994530,11cyclictest23-21ksoftirqd/109:48:421
53248199443,5cyclictest131rcu_preempt11:15:001
532481994428,13cyclictest1-21systemd10:15:001
532481994427,12cyclictest310-21in:imklog12:27:001
532481994427,12cyclictest275-21dbus-daemon11:40:021
532481994427,12cyclictest275-21dbus-daemon11:40:011
532479994427,12cyclictest275-21dbus-daemon07:55:010
532481994331,8cyclictest23-21ksoftirqd/109:40:011
532481994328,12cyclictest23-21ksoftirqd/108:30:021
532481994325,13cyclictest23-21ksoftirqd/110:40:011
53248199425,6cyclictest131rcu_preempt11:20:011
532481994230,9cyclictest1-21systemd11:05:001
532481994228,10cyclictest3467602-21systemd-journal08:25:011
532481994228,10cyclictest3467602-21systemd-journal08:25:011
532481994223,14cyclictest535008-21systemctl08:00:011
532481994223,14cyclictest535008-21systemctl08:00:001
53248199421,37cyclictest23-21ksoftirqd/109:40:031
532481994129,10cyclictest23-21ksoftirqd/111:51:171
532481994126,11cyclictest23-21ksoftirqd/109:30:031
532479994129,7cyclictest309-21in:imuxsock08:00:020
532479994129,7cyclictest309-21in:imuxsock08:00:010
532481994028,9cyclictest23-21ksoftirqd/111:22:381
532481994027,9cyclictest3467602-21systemd-journal11:00:001
532481994027,10cyclictest23-21ksoftirqd/109:22:551
532481994026,11cyclictest23-21ksoftirqd/109:59:481
532481994026,11cyclictest23-21ksoftirqd/109:59:481
532481994024,4cyclictest541827-21sh09:16:281
53248199402,3cyclictest131rcu_preempt12:00:021
532481993930,6cyclictest309-21in:imuxsock11:05:021
532481993927,8cyclictest533963-21cron07:40:011
53248199392,6cyclictest131rcu_preempt10:50:441
532481993926,10cyclictest23-21ksoftirqd/112:10:441
532481993925,9cyclictest309-21in:imuxsock08:41:051
532481993925,10cyclictest23-21ksoftirqd/112:31:341
532481993925,10cyclictest0-21swapper/112:25:011
53248199391,3cyclictest131rcu_preempt07:55:011
532481993831,4cyclictest540976-21latency_hist09:15:021
532481993829,7cyclictest23-21ksoftirqd/109:52:461
532481993828,7cyclictest23-21ksoftirqd/110:44:521
532481993828,7cyclictest23-21ksoftirqd/108:05:001
532481993827,7cyclictest535520-21cron08:10:001
532481993826,9cyclictest536999-21sed08:35:021
532481993826,9cyclictest23-21ksoftirqd/107:25:001
532481993823,11cyclictest23-21ksoftirqd/112:00:181
532481993725,9cyclictest23-21ksoftirqd/112:17:181
532481993721,11cyclictest275-21dbus-daemon09:00:001
532481993627,6cyclictest23-21ksoftirqd/107:45:011
532481993627,6cyclictest23-21ksoftirqd/107:45:011
532481993627,5cyclictest533133-21latency_hist07:20:021
532481993624,9cyclictest23-21ksoftirqd/107:30:011
532481993620,9cyclictest309-21in:imuxsock07:35:001
532481993620,12cyclictest275-21dbus-daemon08:15:011
532481993620,12cyclictest275-21dbus-daemon08:15:011
532481993612,21cyclictest221rcuc/111:28:501
53248199351,4cyclictest131rcu_preempt10:20:521
53248199351,4cyclictest131rcu_preempt10:20:511
532481993424,8cyclictest23-21ksoftirqd/108:40:011
532481993424,8cyclictest23-21ksoftirqd/108:40:001
53248199341,3cyclictest131rcu_preempt10:25:281
532481993224,6cyclictest23-21ksoftirqd/108:55:031
532481993224,6cyclictest23-21ksoftirqd/108:55:021
532481993222,7cyclictest532562-21latency_hist07:10:011
532481993222,7cyclictest23-21ksoftirqd/107:45:411
532481993220,9cyclictest23-21ksoftirqd/108:45:341
53247999301,25cyclictest2383687-1kworker/0:2H07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional