You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-10-20 - 00:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Mon Oct 19, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
973593998982,3cyclictest0-21swapper/011:35:030
973593998720,13cyclictest988217-21ssh09:27:240
973593998528,10cyclictest1026189-21ssh10:47:340
973593998428,13cyclictest1042331-21ssh11:21:360
973593998424,13cyclictest997552-21ssh09:47:090
973593998027,12cyclictest1058378-21ssh11:55:270
973593998027,12cyclictest1058378-21ssh11:55:270
973593997730,42cyclictest16950irq/224-800000010:00:000
973593997630,41cyclictest1-21systemd11:10:010
973593997628,12cyclictest981059-21ssh09:12:290
973593997625,11cyclictest1071467-21ssh12:23:140
973593997524,14cyclictest1068716-21ssh12:17:210
973593997517,18cyclictest16850irq/134-800000011:45:430
973593997517,18cyclictest16850irq/134-800000011:45:430
973593997426,14cyclictest1007235-21ssh10:07:320
97359399742,30cyclictest1013038-21run-parts10:20:010
973593997325,13cyclictest1033833-21ssh11:03:490
973593997322,15cyclictest1020386-21ssh10:35:030
973593997223,12cyclictest1065373-21ssh12:10:050
973593997130,37cyclictest16850irq/134-800000010:43:420
973593997119,11cyclictest1041100-21ssh11:19:140
973593997028,10cyclictest1029543-21sh10:55:010
973593997026,40cyclictest16850irq/134-800000010:35:020
973593997026,40cyclictest16850irq/134-800000010:35:010
973593997024,12cyclictest993038-21ssh09:37:450
973593997020,9cyclictest309-21in:imuxsock09:20:000
97359399692,27cyclictest3467602-21systemd-journal10:14:110
97359399691,4cyclictest999088-21ssh09:50:120
973593996826,11cyclictest1-21systemd10:05:010
97359399682,5cyclictest1-21systemd10:55:480
97359399672,10cyclictest1049451-21ssh11:36:480
97359399672,10cyclictest1049451-21ssh11:36:480
97359399661,28cyclictest1077460-21rm12:35:340
97359399661,28cyclictest1077460-21rm12:35:340
973593996532,29cyclictest16950irq/224-800000009:43:510
973593996527,7cyclictest1056603-21ssh11:51:530
973601996445,15cyclictest3467602-21systemd-journal08:55:012
973601996445,15cyclictest3467602-21systemd-journal08:55:002
973593996432,26cyclictest19350irq/34-mmc109:25:020
973593996414,45cyclictest16850irq/134-800000012:29:020
973593996352,3cyclictest0-21swapper/007:10:020
97359399632,32cyclictest982715-21kworker/u8:309:35:000
973593996319,12cyclictest1060909-21ssh12:00:470
973601996230,7cyclictest1010654-21kworker/2:111:28:222
973593996157,2cyclictest0-21swapper/007:45:030
973593996120,36cyclictest16850irq/134-800000011:26:540
97359399611,43cyclictest1014682-21ssh10:23:170
973601996035,18cyclictest3467602-21systemd-journal08:45:012
973593996026,10cyclictest12-21ksoftirqd/011:10:110
97359399595,50cyclictest107599450ssh12:32:410
973593995926,12cyclictest1017540-21ssh10:29:260
973601995849,5cyclictest977206-21kworker/2:012:38:502
973601995849,5cyclictest977206-21kworker/2:012:38:492
973601995849,5cyclictest977206-21kworker/2:008:35:022
973593995828,2cyclictest0-21swapper/008:41:130
973604995737,14cyclictest275-21dbus-daemon08:40:023
973604995737,14cyclictest275-21dbus-daemon08:40:023
973601995738,15cyclictest1-21systemd07:50:002
973604995641,11cyclictest1031621-21sshd10:59:153
973601995646,6cyclictest733593-21kworker/2:209:52:482
97359599563,49cyclictest1025080-21ssh10:45:031
973593995632,20cyclictest16950irq/224-800000007:10:100
973593995627,6cyclictest1-21systemd11:45:000
973601995547,5cyclictest977206-21kworker/2:011:12:422
973601995546,6cyclictest977206-21kworker/2:012:06:162
973601995546,6cyclictest977206-21kworker/2:012:06:152
973601995544,7cyclictest733593-21kworker/2:209:36:342
973601995542,9cyclictest977206-21kworker/2:009:43:332
973601995446,5cyclictest977206-21kworker/2:010:36:182
973601995445,6cyclictest977206-21kworker/2:011:42:392
973601995445,6cyclictest977206-21kworker/2:010:04:492
973601995444,6cyclictest1010654-21kworker/2:110:25:462
973601995345,5cyclictest977206-21kworker/2:011:46:212
973601995345,5cyclictest977206-21kworker/2:011:46:212
973601995345,5cyclictest1010654-21kworker/2:111:15:552
973601995344,6cyclictest1010654-21kworker/2:111:25:022
973601995344,6cyclictest1010654-21kworker/2:110:31:422
973601995344,6cyclictest1010654-21kworker/2:110:31:422
973601995344,5cyclictest977206-21kworker/2:012:15:042
973601995344,5cyclictest977206-21kworker/2:011:31:022
97360499521,3cyclictest131rcu_preempt11:32:343
973601995244,5cyclictest733593-21kworker/2:209:31:012
973601995244,5cyclictest733593-21kworker/2:209:16:152
973601995243,6cyclictest977206-21kworker/2:009:45:392
973601995243,5cyclictest977206-21kworker/2:012:25:132
973601995243,5cyclictest733593-21kworker/2:209:56:072
973601995242,6cyclictest977206-21kworker/2:010:09:322
973601995241,7cyclictest977206-21kworker/2:010:11:582
973601995230,16cyclictest1-21systemd10:59:592
97359599521,13cyclictest1033647-21ssh11:03:251
973593995212,7cyclictest16850irq/134-800000012:05:460
973593995212,7cyclictest16850irq/134-800000012:05:460
973601995143,5cyclictest977206-21kworker/2:012:20:592
973601995143,5cyclictest977206-21kworker/2:011:51:542
973601995143,5cyclictest977206-21kworker/2:010:41:112
973601995142,6cyclictest977206-21kworker/2:011:56:192
973601995142,6cyclictest977206-21kworker/2:011:56:182
973601995142,5cyclictest977206-21kworker/2:011:07:372
973601995142,5cyclictest733593-21kworker/2:209:29:462
973601995142,5cyclictest733593-21kworker/2:209:21:212
97360199513,45cyclictest979921-21ssh09:10:032
97359599519,10cyclictest23-21ksoftirqd/110:19:211
973595995136,11cyclictest275-21dbus-daemon08:50:001
973595995111,6cyclictest131rcu_preempt12:15:041
97359399511,27cyclictest19350irq/34-mmc108:25:030
973601995042,5cyclictest977206-21kworker/2:012:32:482
973601995042,5cyclictest977206-21kworker/2:011:38:212
973601995042,5cyclictest977206-21kworker/2:011:38:212
973601995042,5cyclictest977206-21kworker/2:010:15:512
973601995042,5cyclictest977206-21kworker/2:008:38:152
973601995042,5cyclictest977206-21kworker/2:008:38:142
973601995041,5cyclictest977206-21kworker/2:011:04:332
973601995040,6cyclictest977206-21kworker/2:010:45:142
973593995018,11cyclictest975184-21sort07:40:010
973604994935,9cyclictest275-21dbus-daemon07:40:003
973601994941,5cyclictest977206-21kworker/2:012:02:452
973601994941,5cyclictest977206-21kworker/2:010:50:332
973601994941,5cyclictest733593-21kworker/2:209:11:442
973601994939,6cyclictest977206-21kworker/2:012:10:282
973593994933,12cyclictest275-21dbus-daemon08:10:010
973593994933,12cyclictest275-21dbus-daemon08:10:000
973593994929,4cyclictest0-21swapper/007:58:400
973604994834,11cyclictest978715-21sshd08:49:153
973604994830,14cyclictest1-21systemd12:10:013
973604994830,14cyclictest1-21systemd12:10:003
97360199489,28cyclictest975084-21modprobe07:40:012
973601994840,5cyclictest733593-21kworker/2:209:00:002
973601994840,5cyclictest1010654-21kworker/2:110:20:472
973601994838,6cyclictest733593-21kworker/2:208:10:012
973601994838,6cyclictest733593-21kworker/2:208:10:012
973601994830,12cyclictest275-21dbus-daemon07:55:002
973601994822,5cyclictest30-21ksoftirqd/207:34:592
973595994832,11cyclictest1-21systemd11:55:011
973595994831,13cyclictest275-21dbus-daemon11:50:001
973595994831,13cyclictest275-21dbus-daemon11:50:001
973595994828,14cyclictest275-21dbus-daemon12:10:011
973595994828,14cyclictest275-21dbus-daemon12:10:001
97359599481,9cyclictest678060-1kworker/u9:009:15:021
973601994740,4cyclictest733593-21kworker/2:207:42:142
973601994739,5cyclictest733593-21kworker/2:208:05:022
973595994737,6cyclictest1060327-21sendmail-msp12:00:011
973595994737,6cyclictest1060327-21sendmail-msp12:00:001
973595994731,12cyclictest275-21dbus-daemon11:15:011
97359599472,3cyclictest131rcu_preempt10:49:361
973595994720,9cyclictest23-21ksoftirqd/112:10:491
973601994638,5cyclictest733593-21kworker/2:209:05:012
973601994638,5cyclictest733593-21kworker/2:207:20:022
973601994624,17cyclictest275-21dbus-daemon07:15:002
97359599466,26cyclictest978946-21kworker/u8:209:00:021
973595994633,10cyclictest275-21dbus-daemon08:00:001
973595994632,11cyclictest1-21systemd08:55:011
973595994632,11cyclictest1-21systemd08:55:001
973595994632,10cyclictest3467602-21systemd-journal11:07:131
973595994627,14cyclictest1-21systemd07:30:021
973595994627,14cyclictest1-21systemd07:30:011
97360499455,8cyclictest37-21ksoftirqd/312:25:563
97360499453,4cyclictest131rcu_preempt09:24:073
97360499452,3cyclictest131rcu_preempt10:48:593
97360499452,3cyclictest131rcu_preempt10:26:103
97360499451,4cyclictest131rcu_preempt12:13:383
973601994537,5cyclictest733593-21kworker/2:207:28:052
973601994537,5cyclictest733593-21kworker/2:207:28:042
973601994511,7cyclictest131rcu_preempt08:20:012
97359599453,3cyclictest131rcu_preempt11:40:021
97359599453,3cyclictest131rcu_preempt11:40:021
97359599452,3cyclictest131rcu_preempt10:35:401
97359399453,4cyclictest131rcu_preempt09:05:020
973593994531,10cyclictest978209-21systemctl08:40:010
973593994531,10cyclictest978209-21systemctl08:40:000
97359399452,36cyclictest979004-21cat08:55:020
97359399452,36cyclictest979004-21cat08:55:010
97359399451,29cyclictest0-21swapper/007:16:480
97360499444,3cyclictest131rcu_preempt11:41:053
97360499443,4cyclictest131rcu_preempt09:27:063
973604994431,8cyclictest1-21systemd10:40:003
973604994428,12cyclictest37-21ksoftirqd/310:55:013
973604994427,12cyclictest37-21ksoftirqd/309:46:463
97360499442,4cyclictest131rcu_preempt09:53:333
97360499442,4cyclictest131rcu_preempt09:31:223
973601994436,5cyclictest733593-21kworker/2:208:13:192
973595994433,8cyclictest996101-21sshd09:44:151
973595994429,12cyclictest1053427-21wc11:45:021
97359599441,3cyclictest131rcu_preempt09:45:031
97359599441,3cyclictest131rcu_preempt09:31:071
973601994335,5cyclictest733593-21kworker/2:207:20:422
973601994334,6cyclictest733593-21kworker/2:207:55:202
97359599439,7cyclictest131rcu_preempt10:02:421
97359599437,6cyclictest977915-21kworker/u8:308:35:001
97359599434,33cyclictest1070363-21kworker/u8:012:30:021
97359599432,3cyclictest131rcu_preempt12:04:041
973593994334,5cyclictest16950irq/224-800000009:10:010
973593994329,9cyclictest3467602-21systemd-journal08:20:000
973604994233,5cyclictest1067769-21sshd12:15:143
973604994232,6cyclictest309-21in:imuxsock08:20:003
973604994231,6cyclictest309-21in:imuxsock08:05:013
973604994227,11cyclictest1015445-21systemd-run10:25:013
973604994227,10cyclictest37-21ksoftirqd/310:00:563
973604994226,12cyclictest3467602-21systemd-journal07:15:013
973601994234,5cyclictest733593-21kworker/2:208:23:352
973601994226,11cyclictest30-21ksoftirqd/208:30:002
97359599423,9cyclictest0-21swapper/109:39:021
973595994232,6cyclictest979769-21diskmemload09:10:011
973595994229,10cyclictest23-21ksoftirqd/110:21:071
97359599421,3cyclictest131rcu_preempt09:25:011
97360499413,3cyclictest131rcu_preempt11:16:473
97360499413,3cyclictest131rcu_preempt09:11:113
973604994131,6cyclictest309-21in:imuxsock11:10:033
973604994129,9cyclictest37-21ksoftirqd/311:07:583
973604994122,13cyclictest275-21dbus-daemon07:25:013
97360499411,3cyclictest131rcu_preempt12:21:483
97360499411,3cyclictest131rcu_preempt11:24:453
973604994113,17cyclictest1013245-21ssh10:20:033
97359599417,6cyclictest131rcu_preempt12:33:571
97359599413,3cyclictest131rcu_preempt12:38:121
97359599413,3cyclictest131rcu_preempt12:38:121
97359599412,9cyclictest1031107-21ssh10:58:091
97359599411,3cyclictest131rcu_preempt10:25:291
973593994126,11cyclictest275-21dbus-daemon08:25:480
973604994031,7cyclictest37-21ksoftirqd/310:12:053
973604994030,7cyclictest979769-21diskmemload12:40:023
973604994030,7cyclictest979769-21diskmemload12:40:013
973604994028,9cyclictest37-21ksoftirqd/308:10:013
973604994028,9cyclictest37-21ksoftirqd/308:10:013
973604994025,11cyclictest37-21ksoftirqd/309:19:033
97360499401,5cyclictest131rcu_preempt07:35:003
973601994033,5cyclictest977206-21kworker/2:008:45:512
97359599409,7cyclictest131rcu_preempt07:20:001
973595994029,9cyclictest23-21ksoftirqd/112:24:011
973595994025,11cyclictest23-21ksoftirqd/110:15:011
973593994014,4cyclictest131rcu_preempt08:05:020
973604993928,8cyclictest284-21systemd-logind11:02:003
973604993927,9cyclictest1060306-21systemctl12:00:003
973604993927,9cyclictest1060306-21systemctl11:59:593
97360499392,3cyclictest131rcu_preempt11:55:023
973604993920,14cyclictest275-21dbus-daemon12:35:013
973604993914,9cyclictest275-21dbus-daemon08:30:013
97359599397,23cyclictest1001152-21kworker/u8:310:00:031
973595993927,9cyclictest23-21ksoftirqd/111:33:471
97359599391,3cyclictest131rcu_preempt10:07:121
97359599391,34cyclictest977424-21sshd08:22:141
973593993932,4cyclictest19350irq/34-mmc108:10:040
973593993924,11cyclictest974566-21sshd07:28:150
973593993924,11cyclictest974566-21sshd07:28:140
973604993824,10cyclictest37-21ksoftirqd/309:40:403
97360499382,3cyclictest131rcu_preempt10:30:513
97360499382,3cyclictest131rcu_preempt10:30:503
973604993818,3cyclictest37-21ksoftirqd/311:26:073
973604993815,17cyclictest275-21dbus-daemon08:45:003
97360499381,3cyclictest131rcu_preempt11:38:153
97360499381,3cyclictest131rcu_preempt11:38:143
97359599389,3cyclictest131rcu_preempt10:30:071
97359599389,3cyclictest131rcu_preempt10:30:071
973595993827,7cyclictest3467602-21systemd-journal08:40:031
973595993826,8cyclictest309-21in:imuxsock08:36:151
973595993826,8cyclictest309-21in:imuxsock08:36:141
973595993824,11cyclictest23-21ksoftirqd/111:27:411
97359399380,36cyclictest0-21swapper/009:00:010
97360499379,5cyclictest131rcu_preempt09:38:053
97360499372,7cyclictest994117-21kworker/3:210:05:213
973604993725,9cyclictest3467602-21systemd-journal07:46:143
973604993725,10cyclictest37-21ksoftirqd/309:56:253
97360499372,4cyclictest131rcu_preempt10:40:263
97360499372,3cyclictest131rcu_preempt12:04:123
973604993721,11cyclictest275-21dbus-daemon09:00:013
97360499371,3cyclictest131rcu_preempt09:05:013
97359599375,3cyclictest131rcu_preempt11:21:591
973595993727,8cyclictest23-21ksoftirqd/108:05:021
973595993725,9cyclictest23-21ksoftirqd/107:45:141
97359599372,3cyclictest131rcu_preempt11:20:001
973595993722,11cyclictest3467602-21systemd-journal07:53:151
973595993721,12cyclictest987677-21ssh09:26:161
973595993721,12cyclictest3467602-21systemd-journal07:24:591
973593993731,4cyclictest16950irq/224-800000008:35:000
973593993728,6cyclictest19350irq/34-mmc107:25:020
973604993625,9cyclictest37-21ksoftirqd/308:55:003
973604993625,9cyclictest37-21ksoftirqd/308:55:003
97360499362,3cyclictest131rcu_preempt11:47:043
97360499362,3cyclictest131rcu_preempt11:47:033
973604993623,9cyclictest37-21ksoftirqd/307:29:423
973604993623,9cyclictest37-21ksoftirqd/307:29:423
973601993629,4cyclictest301ksoftirqd/207:10:012
973595993623,10cyclictest23-21ksoftirqd/107:35:121
97360499354,5cyclictest975836-21sshd07:50:143
973604993527,6cyclictest37-21ksoftirqd/308:25:033
973604993522,9cyclictest37-21ksoftirqd/307:43:113
973595993527,6cyclictest23-21ksoftirqd/107:35:021
973595993526,7cyclictest23-21ksoftirqd/109:15:151
973595993522,9cyclictest1-21systemd07:15:011
973595993522,11cyclictest23-21ksoftirqd/110:52:171
97359399351,17cyclictest0-21swapper/007:50:020
97360499345,8cyclictest37-21ksoftirqd/308:35:003
973604993424,5cyclictest37-21ksoftirqd/307:59:593
97359599349,5cyclictest131rcu_preempt09:51:161
97359599343,3cyclictest131rcu_preempt08:20:011
97359399344,7cyclictest0-21swapper/007:53:480
973593993419,11cyclictest974822-21sshd07:34:140
973604993323,7cyclictest979907-21latency_hist09:10:023
973595993324,5cyclictest977756-21latency_hist08:30:021
973595993321,10cyclictest23-21ksoftirqd/108:15:001
97359399331,29cyclictest0-21swapper/008:45:030
973595993222,8cyclictest23-21ksoftirqd/108:10:021
973595993222,8cyclictest23-21ksoftirqd/108:10:011
973595993219,9cyclictest1-21systemd07:45:011
97360499316,3cyclictest131rcu_preempt07:10:023
97360499313,3cyclictest131rcu_preempt08:15:003
973604993115,11cyclictest309-21in:imuxsock07:16:143
97359599312,4cyclictest131rcu_preempt09:00:001
97359599301,7cyclictest973536-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional