You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 14:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 07, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10017339912637,5cyclictest131rcu_preempt20:45:021
100174099976,30cyclictest0-21swapper/300:10:023
100173699937,5cyclictest131rcu_preempt23:25:012
1001725998028,27cyclictest1028900-21ssh21:53:560
1001725998028,27cyclictest1028900-21ssh21:53:550
1001725997527,16cyclictest1038186-21ssh22:13:110
1001725997527,16cyclictest1038186-21ssh22:13:110
1001736996718,44cyclictest1005341-21wc20:20:012
100173699671,19cyclictest1011275-21ssh21:16:432
1001736996625,36cyclictest1083478-21sh23:47:582
1001736996625,36cyclictest1083478-21sh23:47:572
100173699651,18cyclictest1094893-21ssh00:11:392
100173699651,12cyclictest1009901-21ssh21:14:002
100173699642,19cyclictest1088851-21ssh23:59:152
100173699642,14cyclictest1054340-21ssh22:46:562
100173699642,14cyclictest1054340-21ssh22:46:552
100173699641,17cyclictest48-21kcompactd022:10:152
100173699641,17cyclictest48-21kcompactd022:10:142
100173699632,20cyclictest1050192-21ssh22:38:242
100173699632,12cyclictest1001897-21sshd19:12:502
100173699632,10cyclictest1051195-21ssh22:40:082
100173699631,13cyclictest1057967-21ssh22:54:472
100173699631,10cyclictest48-21kcompactd023:30:292
100173699631,10cyclictest1068026-21ssh23:15:222
1001725996322,31cyclictest16850irq/134-800000022:30:010
100173699622,9cyclictest1023006-21ssh21:41:212
1001736996226,32cyclictest1097974-21sh00:18:142
100173699622,18cyclictest1078565-21ssh23:37:392
100173699622,18cyclictest1078565-21ssh23:37:382
100173699621,8cyclictest1047943-21ssh22:33:452
100173699621,8cyclictest1026458-21ssh21:48:482
100173699621,10cyclictest1084677-21ssh23:50:092
100173699612,14cyclictest1007080-21sshd20:50:252
100173699611,9cyclictest1092120-21ssh00:05:392
100173699611,9cyclictest1018397-21ssh21:31:472
100173699611,8cyclictest1073686-21ssh23:27:282
1001736996116,40cyclictest1039511-21sh22:15:492
100173699611,15cyclictest1028174-21ssh21:52:172
100173699611,15cyclictest1028174-21ssh21:52:162
100173699611,13cyclictest1063699-21sh23:06:202
100173699611,13cyclictest1063699-21sh23:06:202
1001725996119,32cyclictest16850irq/134-800000021:44:150
100173699602,17cyclictest1062825-21cron23:05:002
100173699602,16cyclictest1013873-21ssh21:22:122
100173699601,9cyclictest48-21kcompactd021:35:072
100173699601,8cyclictest1104649-21ssh00:32:132
100173699601,8cyclictest1091584-21cron00:05:012
1001725996019,33cyclictest16850irq/134-800000023:06:000
1001725996019,33cyclictest16850irq/134-800000023:05:590
100173699592,17cyclictest1044369-21ssh22:26:022
100173699592,13cyclictest1016340-21sh21:27:272
1001736995916,39cyclictest1029861-21kworker/u8:122:02:472
1001733995940,14cyclictest275-21dbus-daemon00:00:001
1001733995937,16cyclictest3467602-21systemd-journal19:20:001
1001725995919,32cyclictest16850irq/134-800000022:43:540
1001725995919,31cyclictest16850irq/134-800000023:48:120
1001725995919,31cyclictest16850irq/134-800000023:48:110
1001725995919,31cyclictest16850irq/134-800000023:15:170
1001725995919,31cyclictest16850irq/134-800000023:14:010
1001725995919,31cyclictest16850irq/134-800000021:10:550
1001725995919,31cyclictest16850irq/134-800000000:22:360
100173699582,14cyclictest1102120-21ssh00:26:532
100173699582,11cyclictest1003184-21sshd19:37:032
100173699581,7cyclictest1003476-21sshd19:44:022
100173699581,7cyclictest1003476-21sshd19:44:022
1001736995817,36cyclictest1042085-21sh22:21:062
100173699581,15cyclictest1100666-21ssh00:23:562
100173699581,14cyclictest1067529-21ssh23:14:492
100173699581,14cyclictest1005540-21sendmail-mta20:20:212
1001733995844,10cyclictest0-21swapper/119:10:011
1001725995830,12cyclictest1015526-21ssh21:25:370
1001725995823,16cyclictest131rcu_preempt21:45:490
100173699571,9cyclictest1106502-21ssh00:36:022
100173699571,8cyclictest1058730-21ssh22:56:042
100173699571,8cyclictest1058730-21ssh22:56:032
100173699571,13cyclictest364-21snmpd20:09:302
100172599572,3cyclictest131rcu_preempt23:28:450
100172599571,4cyclictest131rcu_preempt22:18:000
100172599571,3cyclictest131rcu_preempt00:05:390
100173699568,43cyclictest0-21swapper/221:58:472
100173699562,7cyclictest1036561-21munin-run22:10:012
100173699562,13cyclictest1006343-21modprobe20:40:012
1001736995611,40cyclictest1-21systemd20:35:002
100172599564,48cyclictest0-21swapper/021:30:020
1001740995513,10cyclictest131rcu_preempt22:09:593
100173699552,6cyclictest1002472-21munin-run19:25:002
100173699552,16cyclictest1080585-21sh23:41:462
1001736995514,36cyclictest1008047-21sed21:10:032
1001736995511,37cyclictest0-21swapper/219:20:032
100173699551,12cyclictest1002685-21sshd19:27:162
1001725995528,12cyclictest1033107-21ssh22:02:430
1001725995523,15cyclictest1043379-21ssh22:24:100
1001725995522,14cyclictest131rcu_preempt00:16:020
1001725995519,31cyclictest16850irq/134-800000023:35:260
1001725995519,31cyclictest16850irq/134-800000023:35:250
1001725995515,4cyclictest131rcu_preempt00:00:000
100172599551,38cyclictest1104157-21ssh00:31:090
1001736995417,31cyclictest0-21swapper/219:49:592
100173699541,12cyclictest364-21snmpd20:49:002
100173699541,12cyclictest364-21snmpd20:49:002
1001733995445,6cyclictest23-21ksoftirqd/121:50:011
1001725995446,6cyclictest12-21ksoftirqd/021:19:080
100172599543,3cyclictest131rcu_preempt23:50:180
1001725995419,31cyclictest16850irq/134-800000023:20:400
1001725995419,31cyclictest16850irq/134-800000022:34:420
1001725995418,32cyclictest16850irq/134-800000022:51:220
1001725995418,32cyclictest16850irq/134-800000022:36:560
1001725995418,32cyclictest16850irq/134-800000000:03:420
1001725995418,31cyclictest16850irq/134-800000021:20:500
1001725995418,31cyclictest16850irq/134-800000000:25:280
1001725995416,33cyclictest16850irq/134-800000023:34:490
100172599541,3cyclictest131rcu_preempt23:42:400
1001736995319,30cyclictest309-21in:imuxsock19:35:012
100173699531,7cyclictest1007623-21sshd21:02:262
100173699531,11cyclictest364-21snmpd20:11:122
1001725995349,2cyclictest0-21swapper/019:10:020
1001725995332,16cyclictest3467602-21systemd-journal21:00:010
1001725995332,16cyclictest3467602-21systemd-journal21:00:010
1001725995318,31cyclictest16850irq/134-800000022:57:220
1001725995318,31cyclictest16850irq/134-800000022:57:210
1001725995318,31cyclictest16850irq/134-800000022:49:420
1001725995318,31cyclictest16850irq/134-800000022:49:410
1001725995316,32cyclictest16850irq/134-800000021:56:220
100173699522,17cyclictest1004000-21munin-run19:55:002
100173699522,12cyclictest364-21snmpd20:03:292
1001725995226,11cyclictest1094222-21ssh00:10:040
1001725995217,31cyclictest16850irq/134-800000021:38:420
1001725995216,32cyclictest16850irq/134-800000023:02:080
100173699511,15cyclictest3467602-21systemd-journal20:59:262
100173699511,15cyclictest3467602-21systemd-journal20:59:252
100173699511,14cyclictest1-21systemd20:45:002
1001733995135,13cyclictest1060472-21cron23:00:011
1001733995135,13cyclictest1060472-21cron23:00:001
100173399511,6cyclictest131rcu_preempt22:40:011
1001725995114,32cyclictest16850irq/134-800000000:35:170
1001740995029,15cyclictest275-21dbus-daemon00:00:013
1001733995032,14cyclictest1-21systemd22:00:011
1001725995016,30cyclictest16850irq/134-800000022:06:290
100173699491,10cyclictest1001664-21latency_hist19:10:022
1001733994934,12cyclictest1082077-21sh23:45:011
1001733994934,11cyclictest3467602-21systemd-journal23:15:021
1001733994928,14cyclictest275-21dbus-daemon23:55:001
1001733994927,18cyclictest1-21systemd00:10:011
1001725994931,13cyclictest1-21systemd20:45:000
100173699482,17cyclictest275-21dbus-daemon20:00:022
1001733994831,12cyclictest275-21dbus-daemon20:55:011
1001733994828,15cyclictest3467602-21systemd-journal20:30:001
1001725994811,7cyclictest16850irq/134-800000021:10:010
100174099476,5cyclictest131rcu_preempt23:42:063
1001740994736,8cyclictest1100499-21sshd00:23:363
1001740994720,10cyclictest37-21ksoftirqd/321:35:163
100173699471,6cyclictest364-21snmpd20:26:392
1001733994734,10cyclictest1074650-21sshd23:29:351
1001733994731,12cyclictest275-21dbus-daemon23:25:011
1001740994633,10cyclictest3467602-21systemd-journal22:54:593
1001740994611,6cyclictest131rcu_preempt22:14:393
1001740994611,6cyclictest131rcu_preempt22:14:383
1001733994619,18cyclictest275-21dbus-daemon20:50:011
1001733994619,18cyclictest275-21dbus-daemon20:50:011
1001725994633,10cyclictest1-21systemd19:25:010
1001740994531,10cyclictest1060476-21systemctl23:00:013
1001740994531,10cyclictest1060476-21systemctl23:00:013
1001740994530,12cyclictest37-21ksoftirqd/321:55:013
1001740994530,12cyclictest37-21ksoftirqd/321:55:003
100174099451,14cyclictest1054234-21ssh22:46:423
100174099451,14cyclictest1054234-21ssh22:46:423
100173399454,9cyclictest1011849-21ssh21:18:011
1001733994535,7cyclictest23-21ksoftirqd/123:46:431
1001733994535,7cyclictest23-21ksoftirqd/123:46:431
1001733994521,5cyclictest23-21ksoftirqd/121:38:471
100173399451,4cyclictest131rcu_preempt21:12:081
100172599456,4cyclictest131rcu_preempt19:15:000
100174099448,6cyclictest131rcu_preempt21:47:413
100174099445,9cyclictest37-21ksoftirqd/319:31:033
1001740994424,14cyclictest275-21dbus-daemon21:10:003
100174099442,3cyclictest131rcu_preempt23:06:013
100174099442,3cyclictest131rcu_preempt23:06:003
100174099442,36cyclictest1002196-21modprobe19:20:003
100174099441,3cyclictest131rcu_preempt00:04:073
100174099441,38cyclictest3467602-21systemd-journal19:50:003
100173399449,20cyclictest1005284-21modprobe20:20:011
1001733994433,8cyclictest23-21ksoftirqd/123:05:011
1001733994432,9cyclictest23-21ksoftirqd/123:33:251
1001733994431,9cyclictest3467602-21systemd-journal21:03:281
1001733994427,10cyclictest1041331-21sh22:19:591
1001733994426,15cyclictest23-21ksoftirqd/122:02:141
100173399441,3cyclictest131rcu_preempt23:13:291
100172599444,3cyclictest131rcu_preempt19:20:000
100174099439,6cyclictest131rcu_preempt21:29:183
100174099431,5cyclictest131rcu_preempt20:15:003
100174099431,4cyclictest131rcu_preempt23:46:373
100174099431,4cyclictest131rcu_preempt23:46:363
100174099431,4cyclictest131rcu_preempt22:43:093
100173399434,3cyclictest131rcu_preempt21:23:201
1001733994334,5cyclictest1017409-21cron21:30:001
1001733994333,7cyclictest23-21ksoftirqd/122:30:021
1001733994328,11cyclictest23-21ksoftirqd/121:40:291
1001733994319,5cyclictest23-21ksoftirqd/122:21:341
1001725994325,11cyclictest275-21dbus-daemon20:30:010
100174099426,5cyclictest131rcu_preempt22:33:143
100174099425,7cyclictest37-21ksoftirqd/300:30:353
100174099425,5cyclictest131rcu_preempt23:31:583
100174099424,3cyclictest131rcu_preempt22:04:133
1001740994231,7cyclictest1041543-21cat22:20:023
1001740994229,8cyclictest37-21ksoftirqd/321:34:303
1001740994229,10cyclictest37-21ksoftirqd/322:28:043
100174099422,3cyclictest131rcu_preempt23:11:473
100174099421,3cyclictest131rcu_preempt21:15:253
100173399429,18cyclictest1007366-21modprobe21:00:011
100173399429,18cyclictest1007366-21modprobe21:00:001
1001733994229,9cyclictest1-21systemd20:00:011
1001733994227,11cyclictest23-21ksoftirqd/122:51:541
100173399422,23cyclictest1091598-21munin-run00:05:001
1001733994217,18cyclictest1079572-21sh23:40:011
1001733994217,18cyclictest1079572-21sh23:40:011
100173399421,3cyclictest131rcu_preempt22:13:131
100173399421,3cyclictest131rcu_preempt22:13:131
1001725994227,11cyclictest1006299-21sshd20:36:260
100172599422,3cyclictest131rcu_preempt20:05:000
100174099419,5cyclictest131rcu_preempt21:13:043
100174099415,4cyclictest150-21pr/ttyS223:36:373
100174099415,4cyclictest150-21pr/ttyS223:36:363
100174099414,4cyclictest131rcu_preempt23:20:353
100174099413,34cyclictest48-21kcompactd000:15:013
1001740994130,9cyclictest37-21ksoftirqd/322:38:223
1001740994129,9cyclictest275-21dbus-daemon21:00:013
1001740994129,9cyclictest275-21dbus-daemon21:00:003
1001740994127,10cyclictest37-21ksoftirqd/321:24:493
1001740994127,10cyclictest275-21dbus-daemon20:50:013
1001740994127,10cyclictest275-21dbus-daemon20:50:003
100174099412,4cyclictest131rcu_preempt22:25:013
100174099412,4cyclictest131rcu_preempt00:15:043
1001740994121,15cyclictest37-21ksoftirqd/320:45:013
100174099411,3cyclictest131rcu_preempt23:52:133
1001740994112,6cyclictest131rcu_preempt23:03:373
1001733994129,8cyclictest1-21systemd00:35:001
1001733994128,8cyclictest23-21ksoftirqd/121:52:141
1001733994128,8cyclictest23-21ksoftirqd/121:52:131
1001733994126,11cyclictest3467602-21systemd-journal20:02:051
1001733994117,17cyclictest3467602-21systemd-journal20:10:001
100173399411,3cyclictest131rcu_preempt21:30:391
100172599412,5cyclictest1002793-21latency_hist19:30:010
1001725994124,12cyclictest0-21swapper/019:55:000
100172599412,11cyclictest1003666-21latency_hist19:45:020
100172599412,11cyclictest1003666-21latency_hist19:45:020
100174099409,6cyclictest131rcu_preempt23:28:263
100174099406,29cyclictest1002060-21cat19:15:033
1001740994027,9cyclictest275-21dbus-daemon19:25:003
1001740994026,9cyclictest3467602-21systemd-journal19:40:003
100174099402,34cyclictest37-21ksoftirqd/300:30:013
100174099401,3cyclictest131rcu_preempt20:00:023
100173399408,11cyclictest23-21ksoftirqd/100:22:391
100173399403,3cyclictest131rcu_preempt22:48:191
100173399403,3cyclictest131rcu_preempt22:48:181
100173399403,3cyclictest131rcu_preempt00:25:511
1001733994025,11cyclictest275-21dbus-daemon20:25:011
1001733994023,13cyclictest1098842-21sort00:20:011
1001725994029,8cyclictest1004888-21sed20:10:020
1001725994023,12cyclictest275-21dbus-daemon21:05:000
100174099394,3cyclictest131rcu_preempt23:17:103
100174099394,3cyclictest131rcu_preempt00:35:553
100174099394,31cyclictest1004883-21ls20:10:023
100174099392,3cyclictest131rcu_preempt21:55:123
100174099391,3cyclictest131rcu_preempt21:42:583
1001733993923,11cyclictest275-21dbus-daemon22:45:011
1001733993921,14cyclictest1008016-21cut21:10:011
1001733993917,4cyclictest23-21ksoftirqd/122:07:241
100172599393,4cyclictest1003246-21modprobe19:40:010
1001725993916,19cyclictest16850irq/134-800000020:16:120
100172599391,3cyclictest131rcu_preempt19:35:000
100174099387,3cyclictest131rcu_preempt20:28:253
100174099383,4cyclictest131rcu_preempt20:20:013
100173399387,6cyclictest131rcu_preempt19:40:001
100173399383,25cyclictest3467602-21systemd-journal19:30:011
1001733993828,7cyclictest23-21ksoftirqd/123:10:031
1001733993828,7cyclictest23-21ksoftirqd/123:10:021
1001733993824,9cyclictest23-21ksoftirqd/119:55:031
1001725993826,7cyclictest12-21ksoftirqd/020:35:000
100172599382,3cyclictest131rcu_preempt20:15:000
100174099373,4cyclictest131rcu_preempt19:55:013
100174099373,3cyclictest131rcu_preempt20:53:273
100174099373,3cyclictest131rcu_preempt20:25:013
1001740993724,9cyclictest1002826-21sed19:30:023
1001733993727,7cyclictest23-21ksoftirqd/100:13:081
1001733993726,8cyclictest23-21ksoftirqd/100:35:381
1001725993732,3cyclictest16950irq/224-800000019:59:050
100172599371,5cyclictest131rcu_preempt19:46:160
100174099364,3cyclictest131rcu_preempt21:03:273
1001740993623,9cyclictest275-21dbus-daemon20:40:003
1001740993619,12cyclictest275-21dbus-daemon19:45:023
1001740993619,12cyclictest275-21dbus-daemon19:45:013
100174099361,3cyclictest131rcu_preempt20:35:003
1001733993626,5cyclictest23-21ksoftirqd/120:40:001
1001733993624,9cyclictest23-21ksoftirqd/119:45:011
1001733993624,9cyclictest23-21ksoftirqd/119:45:001
100173399361,3cyclictest131rcu_preempt22:31:571
1001725993615,16cyclictest16850irq/134-800000020:47:260
1001725993615,16cyclictest16850irq/134-800000020:47:250
100172599361,32cyclictest0-21swapper/020:21:460
100174099352,4cyclictest131rcu_preempt20:03:053
100172599351,30cyclictest0-21swapper/020:52:220
1001733993424,7cyclictest1003924-21wc19:50:011
1001733993422,8cyclictest23-21ksoftirqd/120:15:011
1001733993326,4cyclictest1003104-21latency_hist19:35:011
1001733993323,5cyclictest23-21ksoftirqd/120:35:021
1001733993321,9cyclictest23-21ksoftirqd/119:23:101
100174099317,4cyclictest131rcu_preempt19:10:013
1001733993015,9cyclictest1002061-21grep19:15:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional