You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 13:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Sep 17, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1059085999475,7cyclictest1142286-21kworker/u8:100:15:020
1059094999122,51cyclictest1083690-21rm21:49:222
1059094998920,37cyclictest1161336-21sh00:35:212
1059094998731,23cyclictest1094190-21sh22:11:362
105909499872,50cyclictest1086766-21ssh21:55:412
1059085998375,5cyclictest12-21ksoftirqd/020:15:020
1059085998375,5cyclictest12-21ksoftirqd/020:15:010
105909499823,40cyclictest1079803-21sh21:40:272
1059094998112,37cyclictest1062797-21latency_hist20:20:022
1059094998112,37cyclictest0-21swapper/223:38:092
105909499772,37cyclictest1121370-21munin-run23:10:012
105909499771,55cyclictest0-21swapper/200:25:022
105909499771,36cyclictest1145505-21ssh00:01:292
105909499762,37cyclictest1150325-21ssh00:11:462
105909499703,34cyclictest0-21swapper/223:43:152
1059094997029,36cyclictest1096737-21sh22:17:122
105909499702,45cyclictest1073147-21sh21:26:362
105909499701,42cyclictest1108882-21ssh22:43:022
105909499701,42cyclictest1108882-21ssh22:43:022
105909499701,40cyclictest1147386-21rm00:05:202
105909499701,40cyclictest1147386-21rm00:05:202
105909499692,39cyclictest3467602-21systemd-journal21:14:192
105909499692,39cyclictest1099628-21rm22:23:212
105909499692,38cyclictest1121611-1kworker/u9:123:12:342
105909499692,38cyclictest1121611-1kworker/u9:123:12:332
1059094996830,34cyclictest1116200-21sh22:58:542
1059094996829,36cyclictest1106131-21sh22:37:032
105909499681,63cyclictest1152523-21ssh00:16:282
105909499681,34cyclictest1065268-21diskmemload21:50:432
1059085996836,29cyclictest12-21ksoftirqd/022:30:010
105909499672,29cyclictest1065268-21diskmemload22:50:422
105909499671,62cyclictest1159977-21ssh00:32:392
1059094996627,36cyclictest1144000-21sh23:58:312
1059094996627,36cyclictest1144000-21sh23:58:312
1059085996619,38cyclictest12-21ksoftirqd/023:15:470
105909499651,61cyclictest1157329-21ssh00:26:562
105909499642,58cyclictest1110555-21ssh22:46:252
1059094996424,36cyclictest1072424-21latency_hist21:25:022
105909499641,59cyclictest1129981-21ssh23:28:152
105909499641,59cyclictest1104874-21ssh22:34:432
105909499641,59cyclictest1091972-21ssh22:06:542
105909499641,59cyclictest1077565-21ssh21:35:502
105909499641,59cyclictest1077565-21ssh21:35:492
105909499641,59cyclictest1068697-21ssh21:17:012
105909499641,58cyclictest1075319-21ssh21:31:012
1059085996420,25cyclictest1074643-21ssh21:30:010
105909499631,59cyclictest1139154-21ssh23:48:052
105909499631,59cyclictest1089306-21ssh22:00:592
105909499631,58cyclictest1142218-21ssh23:54:522
105909499631,58cyclictest1142218-21ssh23:54:522
105909499631,58cyclictest1126479-21ssh23:20:322
105909499631,58cyclictest1101258-21ssh22:26:392
105909499622,56cyclictest1117096-21ssh23:00:292
105909499622,56cyclictest1117096-21ssh23:00:282
105909499621,57cyclictest1132934-21ssh23:34:502
105909499621,57cyclictest1123920-21ssh23:15:032
1059085996226,19cyclictest1093484-21ssh22:10:010
105908599622,43cyclictest1156368-21ssh00:25:000
105909499619,47cyclictest1059273-21cron19:15:002
105909499611,56cyclictest1061901-21munin-run20:05:002
1059085996124,8cyclictest12-21ksoftirqd/021:37:340
1059085996124,8cyclictest12-21ksoftirqd/021:37:330
105908599612,20cyclictest1062672-21grep20:20:000
105909499601,53cyclictest1061582-21sshd19:55:322
1059085996050,6cyclictest16850irq/134-800000000:00:010
1059085996050,6cyclictest16850irq/134-800000000:00:010
1059085996028,16cyclictest1129435-21ssh23:27:030
105909499592,51cyclictest1065029-21munin-run21:05:012
105909499591,54cyclictest1060295-21sshd19:30:332
105909499591,54cyclictest1060295-21sshd19:30:332
105909499582,52cyclictest364-21snmpd20:29:032
105909499582,51cyclictest364-21snmpd19:48:482
105909499572,51cyclictest364-21snmpd20:56:072
105909499572,51cyclictest364-21snmpd20:09:302
105909499571,52cyclictest1060059-21sshd19:29:322
105908999572,51cyclictest1161158-21rm00:35:041
1059085995728,14cyclictest1159344-21ssh00:31:050
105909499563,34cyclictest1065438-21ssh21:10:022
105909499562,49cyclictest1060625-21munin-run19:40:002
105909499562,48cyclictest1063672-21modprobe20:37:322
105908599567,43cyclictest1-21systemd22:35:010
1059085995627,26cyclictest1111956-21ssh22:49:540
1059085995625,5cyclictest131rcu_preempt00:03:130
105909499552,48cyclictest364-21snmpd20:31:072
105909499552,48cyclictest364-21snmpd20:13:312
105909499552,48cyclictest364-21snmpd20:13:302
105909499551,49cyclictest364-21snmpd20:51:312
1059085995529,11cyclictest1097664-21ssh22:19:200
1059085995527,13cyclictest1068946-21ssh21:17:390
1059085995526,13cyclictest1128112-21ssh23:24:220
1059085995525,25cyclictest16850irq/134-800000021:34:140
1059085995525,15cyclictest1153556-21ssh00:18:520
105909499541,49cyclictest364-21snmpd19:20:472
105909499541,49cyclictest1059504-21sshd19:16:332
105909499541,49cyclictest1059504-21sshd19:16:332
1059085995428,19cyclictest1090924-21ssh22:05:010
1059085995428,11cyclictest1157713-21ssh00:27:450
1059085995425,12cyclictest1094178-21ssh22:11:350
105909499532,47cyclictest364-21snmpd20:49:192
105909499532,47cyclictest364-21snmpd20:49:182
105909499532,47cyclictest364-21snmpd19:53:322
105908599532,34cyclictest1149357-21latency_hist00:10:030
105908599532,14cyclictest1109872-21latency_hist22:45:020
105908599532,14cyclictest1109872-21latency_hist22:45:010
105909499522,46cyclictest1060827-21sshd19:40:332
105908999527,24cyclictest1067820-21ssh21:15:031
105908599525,21cyclictest12-21ksoftirqd/021:53:180
1059085995244,5cyclictest16850irq/134-800000022:59:590
1059085995225,11cyclictest1100100-21ssh22:24:200
105908599522,31cyclictest131rcu_preempt00:10:010
105908599521,43cyclictest1105406-21ssh22:35:210
1059085995135,11cyclictest16950irq/224-800000000:40:010
1059085995125,11cyclictest1140244-21ssh23:50:130
1059085995125,11cyclictest1140244-21ssh23:50:130
1059085995123,22cyclictest16850irq/134-800000022:00:010
1059099995019,6cyclictest131rcu_preempt00:00:003
1059099995019,6cyclictest131rcu_preempt00:00:003
1059094995019,26cyclictest275-21dbus-daemon20:45:012
105909499501,44cyclictest1062903-21modprobe20:20:022
1059089995033,12cyclictest3467602-21systemd-journal20:05:001
1059085995041,6cyclictest12-21ksoftirqd/023:45:010
105908599501,43cyclictest1084081-21latency_hist21:50:040
105908999492,3cyclictest131rcu_preempt23:38:221
105908599493,39cyclictest1065234-21kworker/u8:321:10:000
1059085994920,13cyclictest1080125-21ssh21:41:190
105908599491,33cyclictest1133639-21ssh23:36:030
1059085994911,9cyclictest12-21ksoftirqd/023:50:010
1059089994833,11cyclictest1-21systemd22:55:011
105908599485,5cyclictest131rcu_preempt23:02:000
105908599485,5cyclictest131rcu_preempt23:02:000
1059085994829,15cyclictest1123045-21ssh23:13:280
1059085994829,15cyclictest1123045-21ssh23:13:280
105908599482,40cyclictest1060676-21sort19:40:010
105908599481,43cyclictest10713041ssh21:22:390
1059089994731,12cyclictest275-21dbus-daemon22:10:011
1059089994730,13cyclictest23-21ksoftirqd/123:50:341
1059089994730,13cyclictest23-21ksoftirqd/123:50:341
1059089994730,12cyclictest275-21dbus-daemon22:15:001
1059089994728,14cyclictest3467602-21systemd-journal21:10:011
1059085994731,12cyclictest1114412-21rm22:55:020
1059085994731,12cyclictest1059555-21systemctl19:20:010
1059085994731,12cyclictest1059555-21systemctl19:20:000
1059099994629,12cyclictest309-21in:imuxsock21:40:003
1059099994629,12cyclictest309-21in:imuxsock21:40:003
1059089994634,9cyclictest1060308-21sshd19:33:321
1059089994634,9cyclictest1060308-21sshd19:33:311
1059089994625,13cyclictest1126059-21systemctl23:20:001
1059085994631,11cyclictest1121430-21sort23:10:010
1059085994626,14cyclictest275-21dbus-daemon20:55:000
1059099994534,5cyclictest309-21in:imuxsock19:25:013
1059099994525,15cyclictest275-21dbus-daemon00:30:013
1059099994517,19cyclictest37-21ksoftirqd/300:33:583
105908599452,38cyclictest1131733-21ssh23:31:560
105908599451,12cyclictest1065606-21ssh21:10:130
105909999443,3cyclictest131rcu_preempt22:05:023
1059099994432,9cyclictest1128420-21cron23:25:003
1059099994431,10cyclictest3467602-21systemd-journal20:25:003
1059099994429,11cyclictest275-21dbus-daemon00:05:013
1059099994427,11cyclictest275-21dbus-daemon21:55:013
105909999442,3cyclictest131rcu_preempt20:20:003
105909999441,6cyclictest131rcu_preempt23:30:003
105909999441,3cyclictest131rcu_preempt00:19:123
1059089994431,10cyclictest23-21ksoftirqd/121:27:101
1059089994428,12cyclictest3467602-21systemd-journal00:35:321
1059089994426,13cyclictest275-21dbus-daemon20:15:011
1059089994426,13cyclictest275-21dbus-daemon20:15:011
1059089994420,19cyclictest23-21ksoftirqd/122:41:321
1059089994420,19cyclictest23-21ksoftirqd/122:41:321
1059089994412,9cyclictest0-21swapper/123:59:041
1059089994412,9cyclictest0-21swapper/123:59:041
1059085994427,13cyclictest1062929-21sshd20:24:550
105908599442,6cyclictest131rcu_preempt21:00:020
1059085994424,13cyclictest3467602-21systemd-journal20:10:000
1059099994329,10cyclictest1-21systemd00:15:003
1059099994328,10cyclictest3467602-21systemd-journal21:44:593
1059099994325,12cyclictest1067649-21systemd-run21:15:013
105909999432,4cyclictest131rcu_preempt23:36:533
105909999432,3cyclictest131rcu_preempt22:14:513
105909999431,6cyclictest131rcu_preempt22:20:003
105909999431,3cyclictest131rcu_preempt22:44:243
105909999431,3cyclictest131rcu_preempt22:44:243
105909499432,36cyclictest1059203-21cat19:10:022
105908999434,3cyclictest131rcu_preempt21:45:451
105908999433,3cyclictest131rcu_preempt21:50:141
105909999424,5cyclictest37-21ksoftirqd/322:38:213
1059099994234,3cyclictest371ksoftirqd/323:00:003
105909999423,3cyclictest131rcu_preempt23:50:123
105909999423,3cyclictest131rcu_preempt23:50:123
1059099994226,12cyclictest37-21ksoftirqd/322:00:003
1059099994226,12cyclictest3467602-21systemd-journal00:25:003
105909999421,3cyclictest131rcu_preempt21:32:013
105909999421,3cyclictest131rcu_preempt21:10:013
105909999421,3cyclictest131rcu_preempt20:05:023
1059089994234,5cyclictest1154072-21sh00:20:011
1059089994226,11cyclictest3467602-21systemd-journal20:20:001
105908999421,3cyclictest131rcu_preempt21:58:531
105908999421,3cyclictest131rcu_preempt21:39:081
105908999421,3cyclictest131rcu_preempt21:39:081
1059099994131,6cyclictest1072272-21sh21:25:013
105909999412,6cyclictest131rcu_preempt23:45:013
1059099994126,11cyclictest1-21systemd00:10:013
1059099994126,11cyclictest1-21systemd00:10:013
1059099994125,11cyclictest309-21in:imuxsock00:40:003
105909999411,3cyclictest131rcu_preempt23:06:073
105909999411,3cyclictest131rcu_preempt22:26:553
1059089994129,9cyclictest23-21ksoftirqd/122:55:081
1059089994128,10cyclictest1128188-21sshd23:24:321
1059089994128,10cyclictest1081692-21systemd-run21:45:011
1059089994127,11cyclictest3467602-21systemd-journal23:11:331
1059089994127,11cyclictest3467602-21systemd-journal23:11:331
1059089994120,16cyclictest275-21dbus-daemon21:25:001
105908999411,4cyclictest131rcu_preempt23:34:531
105908999411,3cyclictest131rcu_preempt22:45:281
105908599418,5cyclictest131rcu_preempt20:40:010
105908599413,10cyclictest1061505-21cat19:55:010
1059099994030,8cyclictest37-21ksoftirqd/320:55:013
1059099994029,8cyclictest37-21ksoftirqd/322:47:093
105909999402,3cyclictest131rcu_preempt22:06:223
1059089994030,7cyclictest1064710-21sshd20:57:311
1059089994030,5cyclictest309-21in:imuxsock20:30:011
1059089994029,9cyclictest23-21ksoftirqd/100:21:141
1059089994025,11cyclictest23-21ksoftirqd/100:09:281
1059089994025,11cyclictest23-21ksoftirqd/100:09:281
1059089994022,14cyclictest1147054-21sed00:05:011
105908999401,4cyclictest131rcu_preempt22:02:011
105908599402,34cyclictest1063178-21sshd20:29:310
1059085994016,17cyclictest275-21dbus-daemon21:05:010
105909999394,4cyclictest131rcu_preempt22:53:513
1059099993929,8cyclictest37-21ksoftirqd/320:45:023
1059099993926,9cyclictest37-21ksoftirqd/323:05:023
1059099993926,9cyclictest37-21ksoftirqd/323:05:023
1059099993926,9cyclictest37-21ksoftirqd/321:16:353
1059099993924,11cyclictest3467602-21systemd-journal23:35:023
1059099993915,20cyclictest37-21ksoftirqd/323:15:013
1059099993915,20cyclictest37-21ksoftirqd/323:15:013
1059089993931,5cyclictest1063706-21munin-plugin-st20:40:011
1059089993928,7cyclictest23-21ksoftirqd/122:25:021
1059089993927,10cyclictest23-21ksoftirqd/121:30:481
105908999392,3cyclictest131rcu_preempt21:15:481
105908999391,3cyclictest131rcu_preempt22:34:351
105908599392,6cyclictest1060329-21munin-plugin-st19:35:010
105908599392,6cyclictest1060329-21munin-plugin-st19:35:000
105909999388,16cyclictest0-21swapper/323:19:423
1059099993826,10cyclictest37-21ksoftirqd/320:00:003
1059099993824,11cyclictest37-21ksoftirqd/322:21:483
1059099993823,12cyclictest37-21ksoftirqd/321:47:013
1059099993822,11cyclictest275-21dbus-daemon19:15:013
105909999381,3cyclictest131rcu_preempt20:09:593
105908999389,5cyclictest131rcu_preempt00:27:081
1059089993829,5cyclictest1065268-21diskmemload23:10:001
1059089993828,7cyclictest23-21ksoftirqd/123:25:591
1059089993828,7cyclictest1151840-21sed00:15:021
1059089993827,9cyclictest23-21ksoftirqd/122:20:011
1059089993822,9cyclictest23-21ksoftirqd/120:45:021
105908599385,25cyclictest1059096-21latency_hist19:10:000
105909999374,3cyclictest131rcu_preempt22:33:543
1059099993727,6cyclictest309-21in:imuxsock19:20:013
1059099993727,6cyclictest309-21in:imuxsock19:20:013
1059099993725,9cyclictest37-21ksoftirqd/323:46:053
1059099993722,11cyclictest37-21ksoftirqd/319:40:003
1059089993727,8cyclictest23-21ksoftirqd/123:02:321
1059089993727,8cyclictest23-21ksoftirqd/123:02:311
1059089993726,9cyclictest23-21ksoftirqd/122:30:021
1059089993726,8cyclictest23-21ksoftirqd/120:05:561
1059089993725,8cyclictest3467602-21systemd-journal19:55:011
105908599374,27cyclictest1063448-21munin-run20:35:000
105908599372,30cyclictest364-21snmpd19:55:480
1059085993721,12cyclictest1060862-21modprobe19:45:000
105908599371,32cyclictest0-21swapper/019:29:460
105908599371,32cyclictest0-21swapper/019:15:000
1059099993629,4cyclictest1073398-21sh21:27:093
1059099993627,6cyclictest1-21systemd20:40:013
105908999363,3cyclictest131rcu_preempt22:39:301
1059089993628,5cyclictest1136654-21sh23:42:401
1059089993625,8cyclictest1061587-21sshd19:56:311
1059089993624,9cyclictest23-21ksoftirqd/121:05:011
1059089993624,9cyclictest23-21ksoftirqd/120:35:001
1059089993624,9cyclictest23-21ksoftirqd/119:20:011
1059089993624,9cyclictest23-21ksoftirqd/119:20:011
1059089993610,17cyclictest23-21ksoftirqd/123:45:061
1059085993625,8cyclictest12-21ksoftirqd/020:05:020
1059099993528,4cyclictest1061347-21sshd19:54:323
105909999352,11cyclictest150-21pr/ttyS221:04:313
1059089993527,6cyclictest23-21ksoftirqd/119:30:021
105908599351,30cyclictest0-21swapper/020:45:010
105908599351,30cyclictest0-21swapper/019:25:020
1059099993422,9cyclictest1064736-21modprobe21:00:003
1059099993421,10cyclictest1-21systemd20:30:013
105909999341,3cyclictest131rcu_preempt20:15:013
105909999341,3cyclictest131rcu_preempt20:15:013
105908999344,3cyclictest131rcu_preempt20:25:031
105909999334,4cyclictest131rcu_preempt19:49:493
1059099993323,6cyclictest309-21in:imuxsock19:35:013
1059099993323,6cyclictest309-21in:imuxsock19:35:013
1059099993318,11cyclictest37-21ksoftirqd/320:50:023
1059099993318,11cyclictest37-21ksoftirqd/320:50:023
1059089993321,8cyclictest23-21ksoftirqd/119:40:331
105908599335,5cyclictest131rcu_preempt20:45:310
105908599335,5cyclictest131rcu_preempt20:45:310
1059085993320,10cyclictest1061151-21sed19:50:020
1059099993225,4cyclictest1063535-21latency_hist20:35:013
1059099993220,8cyclictest1060840-21sshd19:43:313
1059089993222,6cyclictest23-21ksoftirqd/119:45:521
1059089993218,9cyclictest309-21in:imuxsock19:24:451
1059089993216,12cyclictest0-21swapper/119:40:001
1059099993114,13cyclictest1060046-21sshd19:25:323
105908999314,3cyclictest131rcu_preempt20:50:001
105908999314,3cyclictest131rcu_preempt20:49:591
1059089993122,6cyclictest1064662-21cat20:55:021
1059089993122,6cyclictest1059441-21latency_hist19:15:011
1059089993117,7cyclictest0-21swapper/119:10:011
1059099992922,4cyclictest1059160-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional