You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 21:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 31, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3599367991065,4cyclictest131rcu_preempt12:40:011
3599365991011,86cyclictest150-21pr/ttyS209:05:020
359936799941,89cyclictest0-21swapper/108:30:021
3599365998730,40cyclictest3660368-21ssh11:17:110
3599365998623,45cyclictest3630090-21ssh10:06:320
3599367998321,30cyclictest3605523-21diskmemload10:56:441
359936799792,44cyclictest3648354-21ssh10:49:501
3599365997926,49cyclictest3616074-21ssh09:34:210
3599371997835,38cyclictest3467602-21systemd-journal09:50:022
359936599782,72cyclictest3658681-21ssh11:13:220
359936599771,2cyclictest0-21swapper/009:18:060
359936599762,69cyclictest3633495-21ssh10:14:450
3599365997615,55cyclictest3665091-21sh11:28:110
3599365997522,49cyclictest3666105-21ssh11:30:150
359936599752,17cyclictest3691674-21munin-run12:30:010
3599365997517,52cyclictest16850irq/134-800000012:04:370
3599365997431,38cyclictest3622628-21sh09:49:360
3599365997421,37cyclictest3693251-21ssh12:33:340
359936799732,61cyclictest0-21swapper/108:35:021
3599365997330,39cyclictest3688900-21ssh12:23:280
3599365997321,35cyclictest3625916-21ssh09:56:570
3599365997317,51cyclictest16850irq/134-800000010:21:430
359936599722,13cyclictest3668237-21ssh11:35:140
359936599714,4cyclictest3625085-21ssh09:55:030
359936599711,5cyclictest48-21kcompactd009:20:110
359936599706,55cyclictest16850irq/134-800000012:36:030
3599365997029,25cyclictest3654066-21ssh11:02:380
3599365997029,25cyclictest3654066-21ssh11:02:370
3599365997017,34cyclictest16850irq/134-800000012:13:170
3599365997015,51cyclictest12-21ksoftirqd/010:01:380
359936799691,44cyclictest150-21pr/ttyS209:55:031
3599365996917,32cyclictest16850irq/134-800000010:44:140
3599365996917,32cyclictest16850irq/134-800000010:44:140
359936599691,63cyclictest0-21swapper/010:28:530
3599365996916,34cyclictest16850irq/134-800000011:44:460
3599365996914,50cyclictest3612898-21ssh09:26:430
3599365996914,50cyclictest3612898-21ssh09:26:420
3599365996910,54cyclictest16850irq/134-800000012:16:350
3599365996910,54cyclictest16850irq/134-800000012:16:340
3599365996910,41cyclictest16850irq/134-800000010:19:580
359936599682,61cyclictest16850irq/134-800000009:42:540
3599365996818,34cyclictest3655557-21ssh11:05:550
359936599681,63cyclictest0-21swapper/010:35:010
359936599681,63cyclictest0-21swapper/010:35:010
359936799671,41cyclictest3655972-21ssh11:07:041
359936599676,57cyclictest367264150ssh11:45:260
359936599663,58cyclictest16850irq/134-800000010:45:400
359936599662,59cyclictest366276150ssh11:22:430
3599371996531,30cyclictest3467602-21systemd-journal10:03:332
3599365996520,41cyclictest3678879-21cut12:00:030
3599365996516,31cyclictest3682920-21ssh12:09:390
3599365996516,31cyclictest3682920-21ssh12:09:380
359936599651,55cyclictest16850irq/134-800000011:51:270
3599365996515,33cyclictest16850irq/134-800000010:36:510
3599371996427,5cyclictest3671727-21sh11:43:412
3599371996424,4cyclictest3632720-21sh10:12:452
3599367996426,34cyclictest3679760-21sh12:02:001
359936599641,2cyclictest0-21swapper/010:51:590
359936599641,2cyclictest0-21swapper/010:51:590
3599365996411,49cyclictest3617591-21ssh09:37:460
359936799632,56cyclictest3631439-21munin-run10:10:011
3599367996323,35cyclictest3605523-21diskmemload10:32:211
3599367996323,35cyclictest3605523-21diskmemload10:32:211
3599371996228,29cyclictest3674835-21sh11:50:362
3599371996223,10cyclictest3467602-21systemd-journal10:19:592
3599367996228,30cyclictest3634021-21sh10:15:381
359936799622,55cyclictest3612760-21rm09:26:151
359936799622,55cyclictest3612760-21rm09:26:141
3599367996224,34cyclictest3601622-21latency_hist07:55:021
3599367996224,34cyclictest275-21dbus-daemon11:30:021
359936799622,19cyclictest3691423-21ssh12:29:271
3599371996122,5cyclictest3669750-21sh11:39:102
3599371996122,4cyclictest3636146-21sh10:20:212
3599365996130,26cyclictest3606477-21rm09:11:560
3599371996030,27cyclictest3467602-21systemd-journal09:59:022
3599371996022,5cyclictest3652950-21latency_hist11:00:012
3599367996036,19cyclictest3687316-21cron12:20:011
3599367996036,19cyclictest3687316-21cron12:20:001
359936799603,52cyclictest3660289-21modprobe11:17:001
3599367996029,26cyclictest3611440-21sh09:23:291
359936799602,53cyclictest3662527-21sshd11:22:101
359936799602,52cyclictest3602639-21sshd08:14:091
359936799601,16cyclictest3639918-21sh10:29:371
359936799601,15cyclictest3607249-21sh09:13:531
359936599602,5cyclictest3603996-21munin-run08:40:010
359936599602,18cyclictest3600843-21sh07:40:000
3599371995921,11cyclictest30-21ksoftirqd/207:35:012
3599371995912,14cyclictest275-21dbus-daemon08:40:002
359936799592,5cyclictest3637921-21munin-run10:25:021
359936799591,52cyclictest3666081-21ssh11:30:111
3599365995920,35cyclictest3651165-21sh10:55:520
3599371995828,11cyclictest30-21ksoftirqd/210:06:472
359936799582,5cyclictest3670631-21ssh11:40:491
359936799582,52cyclictest3608097-21ssh09:15:331
359936799582,50cyclictest3617690-21ssh09:38:011
359936799582,12cyclictest3693789-21munin-run12:35:001
359936799581,53cyclictest3653300-21ssh11:00:381
359936799581,53cyclictest3653300-21ssh11:00:371
3599367995815,39cyclictest3674861-21sh11:50:381
359936799581,15cyclictest3604234-21munin-run08:45:001
3599375995731,21cyclictest275-21dbus-daemon10:40:003
3599371995724,5cyclictest3650065-21sh10:53:412
3599371995724,5cyclictest3650065-21sh10:53:402
3599371995724,5cyclictest3604787-21latency_hist08:55:012
3599371995722,9cyclictest30-21ksoftirqd/212:01:312
359936799573,49cyclictest3600823-21sshd07:39:081
359936799572,51cyclictest3684349-21ssh12:12:501
359936799572,50cyclictest3616069-21ssh09:34:201
359936799572,48cyclictest3603451-21/usr/sbin/munin08:30:021
359936799572,48cyclictest3603451-21/usr/sbin/munin08:30:011
3599367995722,30cyclictest3622066-1kworker/u9:109:49:461
3599367995721,31cyclictest1-21systemd08:55:011
359936799571,17cyclictest3625840-21ssh09:56:471
359937199565,40cyclictest3644275-21rm10:40:022
3599371995626,9cyclictest30-21ksoftirqd/212:14:222
3599371995624,6cyclictest3672511-21sh11:45:082
3599371995623,5cyclictest3606328-21sh09:11:342
359936799562,50cyclictest3467602-21systemd-journal11:11:371
359936799562,16cyclictest3678382-21ssh11:59:161
359936799561,8cyclictest3599760-21latency_hist07:15:031
359936799561,8cyclictest3599760-21latency_hist07:15:021
359936599561,51cyclictest274-21cron07:24:590
3599375995535,15cyclictest1-21systemd11:40:013
3599375995534,15cyclictest3467602-21systemd-journal08:40:013
3599371995522,7cyclictest30-21ksoftirqd/209:40:272
3599371995521,10cyclictest30-21ksoftirqd/207:40:182
3599371995517,10cyclictest3638177-21sshd10:25:102
359936799553,48cyclictest3688013-21ssh12:21:021
3599367995530,21cyclictest3619999-21sh09:43:151
359936799552,7cyclictest3682867-21sh12:09:311
359936799552,7cyclictest3682867-21sh12:09:301
359936799552,50cyclictest3628575-21ssh10:03:081
359936799552,48cyclictest3600090-21cut07:24:591
359936799552,47cyclictest3668283-21ssh11:35:211
359936799552,17cyclictest1-21systemd08:20:001
359936799551,49cyclictest3643069-21ssh10:37:171
359936599551,50cyclictest3457945-1kworker/0:0H07:55:020
3599371995436,12cyclictest1-21systemd08:00:002
3599371995424,5cyclictest3661847-21sh11:20:232
3599371995421,7cyclictest3667928-21rm11:35:002
3599371995419,11cyclictest30-21ksoftirqd/210:46:562
359936799542,48cyclictest3632828-21ssh10:13:011
359936799542,47cyclictest3605281-21munin-run09:05:011
359936799542,47cyclictest3601375-21munin-run07:50:011
359936799541,13cyclictest3673704-21sh11:48:181
3599375995324,19cyclictest275-21dbus-daemon09:40:013
3599371995322,7cyclictest30-21ksoftirqd/210:40:532
3599371995322,7cyclictest30-21ksoftirqd/210:40:522
359937199532,21cyclictest3603703-21grep08:35:002
3599371995321,7cyclictest30-21ksoftirqd/211:12:512
359936799532,46cyclictest275-21dbus-daemon08:10:011
359936799532,45cyclictest3605509-21cron09:10:001
359936799532,16cyclictest3600801-21date07:35:031
359936799532,14cyclictest3644658-21sh10:40:501
359936799532,14cyclictest3644658-21sh10:40:491
359936799531,49cyclictest3649260-21ssh10:51:361
359936799531,49cyclictest3649260-21ssh10:51:351
359936799531,17cyclictest3600336-21latency_hist07:30:001
359936599534,9cyclictest3605713-21latency_hist09:10:020
3599375995236,12cyclictest275-21dbus-daemon10:25:013
3599375995231,15cyclictest3467602-21systemd-journal08:50:013
3599371995222,9cyclictest30-21ksoftirqd/208:05:002
3599371995221,10cyclictest30-21ksoftirqd/209:20:562
3599371995217,14cyclictest275-21dbus-daemon08:30:012
3599371995217,14cyclictest275-21dbus-daemon08:30:002
3599365995213,34cyclictest3601124-21/usr/sbin/munin07:45:010
359937599517,30cyclictest3609855-21modprobe09:20:013
3599375995133,13cyclictest275-21dbus-daemon08:35:013
3599371995143,5cyclictest3676992-21kworker/2:012:36:442
3599371995143,5cyclictest3591380-21kworker/2:109:45:542
3599371995124,5cyclictest30-21ksoftirqd/211:57:502
359937199512,44cyclictest1-21systemd11:05:012
359937199512,44cyclictest1-21systemd11:05:002
3599371995122,6cyclictest30-21ksoftirqd/212:06:512
3599371995122,6cyclictest30-21ksoftirqd/212:06:502
359936599512,7cyclictest3601844-21sshd07:56:100
359936599512,7cyclictest3599800-21cron07:20:010
359936599512,7cyclictest3599800-21cron07:20:000
359936599512,6cyclictest3602143-21cron08:05:000
359936599511,6cyclictest0-21swapper/007:47:090
359936599511,15cyclictest3604990-21latency_hist09:00:030
359937599503,4cyclictest131rcu_preempt09:40:453
3599371995042,5cyclictest3588854-21kworker/2:009:28:412
3599371995042,5cyclictest3588854-21kworker/2:009:28:412
359936799502,43cyclictest364-21snmpd08:57:151
359936799502,42cyclictest3603212-21unin-run08:25:001
359936599502,6cyclictest3467602-21systemd-journal07:15:000
3599365995022,22cyclictest12-21ksoftirqd/008:20:000
359936599501,14cyclictest3600555-21sshd07:30:090
3599375994933,9cyclictest3602155-21munin-plugin-st08:05:003
3599371994941,5cyclictest3663622-21kworker/2:011:27:062
3599371994925,9cyclictest30-21ksoftirqd/207:20:022
3599371994925,9cyclictest30-21ksoftirqd/207:20:012
3599371994925,6cyclictest30-21ksoftirqd/209:35:022
359936799491,4cyclictest3599540-21modprobe07:14:591
359936799491,44cyclictest364-21snmpd08:49:421
359936799491,13cyclictest3599423-21cat07:10:011
359936599492,7cyclictest1-21systemd08:45:020
3599375994832,13cyclictest37-21ksoftirqd/312:35:073
3599375994831,9cyclictest309-21in:imuxsock11:50:013
3599375994824,17cyclictest3467602-21systemd-journal07:20:013
3599375994824,17cyclictest3467602-21systemd-journal07:20:003
3599371994831,12cyclictest275-21dbus-daemon08:10:022
359936799487,37cyclictest0-21swapper/108:03:101
359936599482,8cyclictest1-21systemd08:50:010
359936599482,8cyclictest1-21systemd08:35:010
359936599481,11cyclictest1-21systemd07:30:010
3599375994732,10cyclictest37-21ksoftirqd/311:50:353
3599375994730,13cyclictest3627375-21sed10:00:023
3599371994724,5cyclictest3660972-21sh11:18:462
359936599479,32cyclictest0-21swapper/008:25:010
359936599472,7cyclictest3467602-21systemd-journal08:28:450
359936599472,7cyclictest3467602-21systemd-journal08:28:450
359936599472,12cyclictest3599485-21sed07:10:010
359937599469,11cyclictest3659631-1kworker/u9:211:20:273
359937599464,31cyclictest1-21systemd07:50:013
3599375994629,13cyclictest275-21dbus-daemon10:45:013
3599375994629,13cyclictest275-21dbus-daemon10:45:003
3599375994629,12cyclictest275-21dbus-daemon12:15:003
3599375994617,20cyclictest275-21dbus-daemon11:15:013
3599371994638,5cyclictest3676992-21kworker/2:012:26:492
3599371994637,6cyclictest3676992-21kworker/2:012:20:012
3599371994637,6cyclictest3676992-21kworker/2:012:20:002
359936599461,9cyclictest3602381-21sshd08:07:100
359936599461,8cyclictest3602626-21sshd08:11:080
359936599461,13cyclictest364-21snmpd08:52:300
359937599453,4cyclictest131rcu_preempt10:50:383
359937599453,4cyclictest131rcu_preempt10:50:373
3599375994531,11cyclictest37-21ksoftirqd/311:44:283
3599375994524,15cyclictest3467602-21systemd-journal08:20:003
3599375994522,16cyclictest3467602-21systemd-journal12:10:013
3599375994522,16cyclictest3467602-21systemd-journal12:10:003
3599371994537,5cyclictest3654990-21kworker/2:311:09:072
3599371994537,5cyclictest3588854-21kworker/2:010:33:552
3599371994537,5cyclictest3588854-21kworker/2:010:33:552
3599371994537,5cyclictest3588854-21kworker/2:009:36:122
359936799451,41cyclictest275-21dbus-daemon08:00:001
3599375994432,10cyclictest37-21ksoftirqd/308:30:023
3599375994432,10cyclictest37-21ksoftirqd/308:30:013
3599375994431,8cyclictest3600846-21sendmail07:40:003
3599375994427,9cyclictest309-21in:imuxsock09:50:003
359937599442,3cyclictest131rcu_preempt11:05:123
3599375994414,16cyclictest3467602-21systemd-journal12:20:013
3599375994414,16cyclictest3467602-21systemd-journal12:20:013
359936799442,37cyclictest364-21snmpd07:43:351
3599375994329,8cyclictest309-21in:imuxsock11:30:003
3599375994326,12cyclictest275-21dbus-daemon12:05:013
3599375994323,14cyclictest3691659-21cron12:30:013
3599375994317,5cyclictest37-21ksoftirqd/310:01:333
3599371994335,5cyclictest3676992-21kworker/2:012:23:472
3599375994229,8cyclictest3605523-21diskmemload10:10:003
3599371994234,5cyclictest3676992-21kworker/2:012:32:462
3599371994234,5cyclictest3591380-21kworker/2:108:20:012
3599371994227,12cyclictest30-21ksoftirqd/209:10:022
359937599416,23cyclictest3678701-21modprobe12:00:013
3599375994127,10cyclictest37-21ksoftirqd/310:14:453
3599371994128,8cyclictest275-21dbus-daemon08:25:012
3599371994126,11cyclictest275-21dbus-daemon08:50:002
3599375994027,10cyclictest37-21ksoftirqd/309:29:353
3599375994027,10cyclictest37-21ksoftirqd/309:29:353
3599375994026,9cyclictest3605522-21latency_hist09:10:023
3599375994026,11cyclictest37-21ksoftirqd/310:47:063
3599375994026,11cyclictest37-21ksoftirqd/310:33:233
3599375994026,11cyclictest37-21ksoftirqd/310:33:223
3599375994025,11cyclictest37-21ksoftirqd/309:13:193
3599375994024,12cyclictest37-21ksoftirqd/309:20:283
359937599401,4cyclictest131rcu_preempt12:23:123
3599375994014,18cyclictest275-21dbus-daemon07:30:003
359937599396,7cyclictest37-21ksoftirqd/310:19:293
3599375993928,8cyclictest37-21ksoftirqd/308:45:023
3599375993925,11cyclictest37-21ksoftirqd/309:54:213
3599375993924,11cyclictest3467602-21systemd-journal07:42:313
3599375993921,13cyclictest3693129-21sshd12:33:163
3599375993912,5cyclictest131rcu_preempt11:00:003
359937599385,4cyclictest131rcu_preempt10:28:533
3599371993827,9cyclictest30-21ksoftirqd/207:25:022
3599371993825,9cyclictest30-21ksoftirqd/209:15:552
359937599379,23cyclictest1-21systemd09:00:013
359937599373,3cyclictest131rcu_preempt11:02:073
359937599373,3cyclictest131rcu_preempt11:02:063
3599375993726,9cyclictest37-21ksoftirqd/309:34:503
3599375993726,7cyclictest37-21ksoftirqd/311:15:363
3599375993725,9cyclictest37-21ksoftirqd/311:31:513
3599375993724,8cyclictest3467602-21systemd-journal07:14:593
3599375993722,8cyclictest309-21in:imuxsock08:55:003
3599371993726,7cyclictest3467602-21systemd-journal09:05:002
359937599363,3cyclictest131rcu_preempt08:00:013
3599375993621,11cyclictest37-21ksoftirqd/307:55:023
3599375993616,14cyclictest3467602-21systemd-journal07:25:003
3599371993620,12cyclictest1-21systemd08:15:012
3599375993521,9cyclictest3602415-21true08:10:013
3599375993521,11cyclictest37-21ksoftirqd/308:25:013
3599371993528,5cyclictest3600381-21latency_hist07:30:022
3599371993524,8cyclictest30-21ksoftirqd/208:40:102
359937599345,19cyclictest3602660-21munin-run08:15:013
3599375993425,7cyclictest37-21ksoftirqd/307:35:033
3599371993422,9cyclictest309-21in:imuxsock09:00:022
3599371993419,10cyclictest3600843-21cron07:40:012
359937599334,3cyclictest131rcu_preempt09:02:103
359937199334,4cyclictest131rcu_preempt07:10:012
3599371993316,12cyclictest275-21dbus-daemon07:55:022
3599371993118,8cyclictest30-21ksoftirqd/207:50:022
3599371993020,8cyclictest30-21ksoftirqd/207:15:002
3599375992920,6cyclictest3599436-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional