You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-04 - 08:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon May 04, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
127697499761,56cyclictest0-21swapper/223:35:022
127696999761,44cyclictest1330554-21ssh22:50:041
127697799759,44cyclictest48-21kcompactd022:35:023
127696999741,40cyclictest1377123-21ssh00:28:371
1276966997361,3cyclictest0-21swapper/019:25:020
127697799714,28cyclictest1380024-21ssh00:35:013
1276966997046,20cyclictest1376589-21sshd00:27:190
1276977996924,11cyclictest37-21ksoftirqd/322:35:083
1276977996924,11cyclictest37-21ksoftirqd/322:35:083
127696999691,64cyclictest1312356-21ssh22:11:401
127696999691,45cyclictest1283131-21diskmemload21:32:531
1276966996928,9cyclictest1282079-21cron20:50:010
1276974996814,21cyclictest1353954-21munin-run23:40:002
127696999681,36cyclictest1333720-21ssh22:57:011
127696999681,36cyclictest1333720-21ssh22:57:011
1276969996725,37cyclictest1328707-21sh22:46:151
1276969996725,37cyclictest1328707-21sh22:46:151
127696999671,62cyclictest1345579-21ssh23:22:011
127696999671,61cyclictest1308969-21ssh22:04:371
1276977996622,5cyclictest1344701-21sh23:20:023
1276974996613,15cyclictest1299459-21ssh21:44:272
127696999662,60cyclictest1336709-21ssh23:03:181
127696999662,60cyclictest1303222-21ssh21:52:121
127696999661,62cyclictest1378319-21ssh00:30:531
127696999661,61cyclictest1343321-21ssh23:17:171
127696999661,61cyclictest1340699-21ssh23:11:401
127696999661,61cyclictest1339517-21ssh23:09:301
127696999661,61cyclictest1306240-21ssh21:58:501
127696999656,33cyclictest1314201-21rm22:15:331
127696999652,39cyclictest1381161-21ssh00:37:091
127696999651,60cyclictest1374662-21ssh00:23:231
127696999651,60cyclictest1370940-21ssh00:15:251
127696999651,60cyclictest1370213-21ssh00:14:111
127696999651,60cyclictest1362041-21ssh23:56:571
127696999651,60cyclictest1359587-21ssh23:51:401
127696999651,60cyclictest1359587-21ssh23:51:401
127696999651,60cyclictest1355175-21ssh23:42:261
127696999651,60cyclictest1352582-21ssh23:37:001
127696999651,60cyclictest1350552-21ssh23:32:421
127696999651,60cyclictest1327447-21ssh22:43:431
127696999651,60cyclictest1320399-21ssh22:28:541
127696999651,60cyclictest1310194-21ssh22:06:531
127696999651,60cyclictest1298707-21ssh21:42:431
127696999651,60cyclictest1295470-21ssh21:35:461
127696999651,60cyclictest1287493-21ssh21:19:041
127696999651,59cyclictest1324375-21ssh22:37:121
127696999651,59cyclictest1324375-21ssh22:37:121
1276977996423,11cyclictest37-21ksoftirqd/321:00:013
1276977996418,7cyclictest1285562-21cron21:15:003
1276977996418,7cyclictest1285562-21cron21:14:593
127696999641,60cyclictest1367593-21ssh00:08:381
127696999641,59cyclictest1356700-21ssh23:45:201
127696999641,59cyclictest1348902-21ssh23:29:161
127696999641,59cyclictest1317758-21ssh22:23:111
127696999641,59cyclictest1299985-21ssh21:45:061
127696999641,59cyclictest1289799-21ssh21:24:011
1276977996322,5cyclictest1287336-21sh21:18:443
127696999631,59cyclictest1321124-21ssh22:30:071
127696999631,59cyclictest1292516-21ssh21:29:541
127696999631,58cyclictest1364390-21ssh00:01:421
127696999631,58cyclictest1283468-21ssh21:10:131
127696999631,58cyclictest1283468-21ssh21:10:131
1276966996350,8cyclictest16850irq/134-800000021:35:020
127696699633,11cyclictest1-21systemd00:10:010
1276966996312,33cyclictest1292711-21latency_hist21:30:020
1276977996220,5cyclictest1302036-21munin-plugin-st21:50:003
127697499629,14cyclictest1286198-21ssh21:16:022
127696999621,56cyclictest1283107-21sshd21:08:091
1276966996252,5cyclictest16950irq/224-800000000:00:000
1276966996243,14cyclictest1278463-21systemctl19:40:000
1276966996235,20cyclictest16850irq/134-800000022:20:000
1276977996127,10cyclictest37-21ksoftirqd/323:36:533
1276977996123,7cyclictest37-21ksoftirqd/323:20:133
1276974996112,15cyclictest1357448-21ssh23:47:052
127696999612,33cyclictest364-21snmpd20:44:551
127696999611,56cyclictest1279735-21sshd20:02:031
127696999611,56cyclictest1279735-21sshd20:02:031
127696999611,56cyclictest1278962-21sshd19:49:091
127696999611,56cyclictest1278962-21sshd19:49:091
127696999611,54cyclictest1280802-21cron20:25:001
1276966996123,13cyclictest1323217-21munin-run22:35:010
1276977996025,7cyclictest1278694-21sshd19:40:093
1276977996023,12cyclictest37-21ksoftirqd/322:44:283
1276977996023,11cyclictest37-21ksoftirqd/322:50:303
1276977996022,4cyclictest309-21in:imuxsock00:38:193
1276974996017,16cyclictest1336177-21ssh23:02:122
1276974996014,14cyclictest1320014-21ssh22:28:052
1276974996011,15cyclictest1291006-21ssh21:26:212
127696999602,53cyclictest364-21snmpd19:21:181
127696999601,53cyclictest48-21kcompactd020:31:301
1276966996053,5cyclictest12-21ksoftirqd/019:45:020
1276966996053,5cyclictest12-21ksoftirqd/019:45:010
1276966996038,19cyclictest12-21ksoftirqd/000:20:570
1276977995918,8cyclictest1304606-21cat21:55:033
127697499599,14cyclictest1323439-21ssh22:35:032
127697499599,14cyclictest1323439-21ssh22:35:032
1276969995946,7cyclictest0-21swapper/119:10:011
1276969995946,7cyclictest0-21swapper/119:10:011
1276969995934,21cyclictest1282091-21munin-plugin-st20:50:011
127696999591,54cyclictest364-21snmpd20:50:251
127696999591,54cyclictest1282851-21sshd21:02:131
127696999591,54cyclictest1279451-21sshd19:56:091
127696999591,53cyclictest364-21snmpd20:27:111
1276966995927,15cyclictest1320834-21ssh22:30:000
1276977995825,10cyclictest37-21ksoftirqd/321:57:433
1276977995824,4cyclictest1296924-21sh21:39:093
1276977995823,4cyclictest1280855-21latency_hist20:25:013
1276977995821,6cyclictest309-21in:imuxsock23:53:223
1276977995821,6cyclictest309-21in:imuxsock23:53:213
1276977995817,11cyclictest37-21ksoftirqd/322:05:023
1276974995815,11cyclictest30-21ksoftirqd/200:01:212
127696999581,53cyclictest1279985-21sshd20:07:031
127696999581,52cyclictest48-21kcompactd020:11:311
127696999581,51cyclictest48-21kcompactd019:16:031
1276966995828,13cyclictest1284474-21ssh21:12:260
1276966995828,13cyclictest1284474-21ssh21:12:260
127697799573,18cyclictest1377718-21grep00:30:003
1276977995720,8cyclictest37-21ksoftirqd/323:05:073
127697499578,19cyclictest1328464-21ssh22:45:442
127697499578,19cyclictest1328464-21ssh22:45:442
1276974995711,15cyclictest1318121-21ssh22:24:122
127696999572,50cyclictest1279247-21munin-run19:55:011
127696999571,52cyclictest1277922-21sshd19:27:091
1276966995736,17cyclictest16850irq/134-800000000:19:120
1276977995640,12cyclictest275-21dbus-daemon00:00:003
1276977995634,16cyclictest275-21dbus-daemon20:35:013
1276977995622,8cyclictest37-21ksoftirqd/320:03:423
1276977995622,8cyclictest37-21ksoftirqd/320:03:423
1276977995622,7cyclictest37-21ksoftirqd/321:41:243
127697499567,18cyclictest1346441-21ssh23:24:002
1276974995617,8cyclictest30-21ksoftirqd/200:18:312
1276974995610,17cyclictest1277886-21latency_hist19:25:022
127696999562,50cyclictest3467602-21systemd-journal19:33:051
127696999562,50cyclictest3467602-21systemd-journal19:33:051
127696999561,51cyclictest364-21snmpd19:35:111
127696999561,51cyclictest364-21snmpd19:10:051
127696999561,50cyclictest48-21kcompactd020:18:111
127696999561,50cyclictest48-21kcompactd019:40:271
127696999561,49cyclictest48-21kcompactd020:35:561
127696699563,18cyclictest1313738-21true22:15:010
127696699563,16cyclictest1283117-21cron21:10:000
1276966995625,16cyclictest1343441-21ssh23:17:330
127696699562,48cyclictest1332736-21latency_hist22:55:010
127696699561,3cyclictest131rcu_preempt21:46:500
1276977995524,10cyclictest37-21ksoftirqd/300:25:013
1276977995522,10cyclictest37-21ksoftirqd/322:26:063
1276977995521,8cyclictest37-21ksoftirqd/323:01:523
1276974995545,6cyclictest1349292-21kworker/2:000:00:002
1276974995511,15cyclictest1381822-21ssh00:38:512
1276974995511,14cyclictest1327794-21sh22:44:382
1276977995424,9cyclictest37-21ksoftirqd/323:45:163
1276977995417,12cyclictest37-21ksoftirqd/322:12:043
127697499547,13cyclictest1337692-21ssh23:05:042
1276974995437,13cyclictest275-21dbus-daemon19:35:012
1276974995437,13cyclictest275-21dbus-daemon19:35:012
1276974995415,9cyclictest30-21ksoftirqd/223:25:242
1276974995415,9cyclictest30-21ksoftirqd/200:26:362
1276974995415,6cyclictest30-21ksoftirqd/223:11:432
1276974995415,14cyclictest1300657-21ssh21:46:402
127696999542,48cyclictest3467602-21systemd-journal20:55:021
127696699548,38cyclictest0-21swapper/020:40:000
1276966995426,12cyclictest1285968-21ssh21:15:330
1276966995425,14cyclictest1357572-21ssh23:47:330
127696699542,3cyclictest131rcu_preempt22:07:320
1276966995421,30cyclictest12-21ksoftirqd/022:04:000
1276977995326,4cyclictest1368507-21sh00:10:163
1276977995321,6cyclictest37-21ksoftirqd/322:25:033
1276974995319,13cyclictest1284128-21ssh21:11:432
1276974995319,13cyclictest1284128-21ssh21:11:432
1276974995316,6cyclictest30-21ksoftirqd/221:38:452
1276974995314,8cyclictest30-21ksoftirqd/221:23:452
1276966995338,10cyclictest1282584-21cron21:00:000
1276966995330,17cyclictest1378884-21sshd00:32:190
1276966995328,10cyclictest1337255-21ssh23:04:380
1276966995324,13cyclictest1352341-21ssh23:36:250
1276966995320,29cyclictest12-21ksoftirqd/022:55:510
1276966995320,29cyclictest12-21ksoftirqd/022:55:510
1276977995220,9cyclictest37-21ksoftirqd/321:22:493
1276977995212,12cyclictest309-21in:imuxsock20:40:013
1276974995217,7cyclictest1359097-21ssh23:50:282
1276974995217,7cyclictest1359097-21ssh23:50:272
1276966995232,13cyclictest1356397-21latency_hist23:45:020
1276966995230,15cyclictest3467602-21systemd-journal22:20:020
1276966995222,15cyclictest1296663-21ssh21:38:250
1276966995221,26cyclictest1348323-21sh23:27:550
1276977995121,9cyclictest37-21ksoftirqd/322:08:053
1276974995115,7cyclictest1374707-21ssh00:23:282
1276974995114,6cyclictest30-21ksoftirqd/222:12:472
1276966995143,5cyclictest16950irq/224-800000000:40:000
1276966995132,15cyclictest16850irq/134-800000023:53:210
1276966995132,15cyclictest16850irq/134-800000023:53:210
1276966995124,12cyclictest1346890-21ssh23:24:570
1276966995121,6cyclictest1326067-21sh22:40:400
127696699512,14cyclictest1365934-21latency_hist00:05:030
1276977995025,6cyclictest37-21ksoftirqd/322:56:423
1276977995025,6cyclictest37-21ksoftirqd/322:56:423
127697499509,10cyclictest1331147-21ssh22:51:342
1276974995042,5cyclictest1268615-21kworker/2:200:05:102
1276974995033,14cyclictest1309963-21ssh22:06:242
1276974995033,13cyclictest1315716-21ssh22:19:072
1276974995031,15cyclictest1379124-21ssh00:33:012
127696699503,9cyclictest1-21systemd23:10:010
1276966995035,11cyclictest275-21dbus-daemon20:30:010
1276966995023,12cyclictest1341384-21ssh23:13:210
1276974994936,10cyclictest1370566-21ssh00:14:562
1276974994934,10cyclictest1279998-21sshd20:09:082
1276974994933,12cyclictest1354997-21ssh23:41:532
1276974994932,14cyclictest1-21systemd21:35:012
1276974994932,13cyclictest1333904-21ssh22:57:252
1276974994932,13cyclictest1333904-21ssh22:57:252
1276974994931,14cyclictest1309048-21ssh22:04:472
1276966994923,11cyclictest1368943-21ssh00:11:110
1276974994834,10cyclictest275-21dbus-daemon20:40:012
1276974994833,12cyclictest1321655-21ssh22:31:242
1276974994833,11cyclictest1280250-21sshd20:14:092
1276974994832,13cyclictest1342510-21ssh23:15:252
1276974994831,14cyclictest1302503-21ssh21:50:312
1276974994830,13cyclictest3467602-21systemd-journal19:45:002
1276974994828,16cyclictest1283112-21sshd21:09:092
1276966994833,12cyclictest1-21systemd21:55:010
1276966994830,13cyclictest3467602-21systemd-journal19:20:000
1276966994821,12cyclictest1306495-21ssh21:59:210
1276977994730,13cyclictest1-21systemd20:55:003
1276974994734,10cyclictest1304677-21ssh21:55:072
127696699474,9cyclictest1351688-21cat23:35:020
1276966994727,16cyclictest1297639-21ssh21:40:060
1276966994725,6cyclictest1329361-21ssh22:47:520
1276966994725,6cyclictest1329361-21ssh22:47:520
1276966994721,5cyclictest3468525-21sendmail-mta22:39:310
1276966994721,5cyclictest3468525-21sendmail-mta22:39:310
1276974994632,11cyclictest1282063-21sshd20:47:092
1276974994630,12cyclictest1278433-21sshd19:39:102
1276966994631,11cyclictest3467602-21systemd-journal19:33:090
1276966994631,11cyclictest3467602-21systemd-journal19:33:090
127696699461,26cyclictest131rcu_preempt21:21:410
1276974994532,9cyclictest1-21systemd19:50:012
1276974994532,9cyclictest1-21systemd19:50:012
1276974994531,11cyclictest1282600-21cron21:00:002
1276966994532,8cyclictest309-21in:imuxsock20:10:000
1276977994419,9cyclictest37-21ksoftirqd/322:18:553
1276974994431,10cyclictest1279247-21munin-run19:55:012
127696699447,25cyclictest1280526-21modprobe20:20:010
1276966994432,9cyclictest12-21ksoftirqd/020:45:020
1276966994424,12cyclictest1278729-21modprobe19:45:000
127697799432,38cyclictest37-21ksoftirqd/322:46:213
127697799432,38cyclictest37-21ksoftirqd/322:46:213
1276974994330,10cyclictest1282344-21munin-run20:55:002
1276974994329,10cyclictest1281060-21munin-run20:30:002
1276974994320,19cyclictest1277960-21systemd-run19:30:012
1276977994226,10cyclictest37-21ksoftirqd/320:10:023
1276977994224,13cyclictest37-21ksoftirqd/300:06:573
1276977994210,9cyclictest131rcu_preempt00:02:263
1276974994229,10cyclictest364-21snmpd19:15:062
1276974994228,10cyclictest1281296-21cron20:35:002
1276966994231,8cyclictest12-21ksoftirqd/019:25:000
1276977994128,10cyclictest37-21ksoftirqd/323:43:073
1276977994127,11cyclictest3467602-21systemd-journal19:35:003
1276977994127,11cyclictest3467602-21systemd-journal19:35:003
1276977994125,12cyclictest1282848-21sshd21:02:093
1276977994125,10cyclictest275-21dbus-daemon19:15:013
1276977994124,14cyclictest1292600-21munin-run21:30:013
1276977994123,12cyclictest1277429-21systemctl19:20:003
1276974994124,13cyclictest364-21snmpd20:18:392
1276974994112,24cyclictest1279873-21cut20:05:022
1276974994112,24cyclictest1279873-21cut20:05:022
127696699415,26cyclictest1281312-21grep20:35:010
1276977994026,11cyclictest37-21ksoftirqd/323:31:453
1276974994026,11cyclictest30-21ksoftirqd/221:05:002
127696699402,31cyclictest1282337-21modprobe20:55:000
1276977993927,9cyclictest37-21ksoftirqd/300:17:003
1276977993926,11cyclictest37-21ksoftirqd/323:11:423
1276977993925,10cyclictest37-21ksoftirqd/321:32:473
1276974993925,10cyclictest30-21ksoftirqd/219:15:012
1276966993931,4cyclictest0-21swapper/020:00:000
1276977993825,10cyclictest1279458-21sshd19:57:093
1276977993822,12cyclictest37-21ksoftirqd/321:10:013
127697799381,10cyclictest1347889-21ssh23:26:463
1276966993829,6cyclictest12-21ksoftirqd/021:05:020
1276977993728,5cyclictest309-21in:imuxsock20:30:013
1276977993723,12cyclictest37-21ksoftirqd/320:15:023
1276966993730,4cyclictest1-21systemd20:05:010
1276966993730,4cyclictest1-21systemd20:05:010
1276966993728,4cyclictest12-21ksoftirqd/019:55:000
127696699372,5cyclictest1277169-21munin-run19:15:010
127696699371,32cyclictest0-21swapper/020:15:020
1276977993627,7cyclictest37-21ksoftirqd/320:50:013
1276974993622,9cyclictest364-21snmpd20:21:032
127697799356,7cyclictest37-21ksoftirqd/319:35:103
127697799355,21cyclictest1280268-21grep20:14:593
1276977993525,7cyclictest37-21ksoftirqd/319:28:553
1276974993523,8cyclictest1279491-21munin-plugin-st20:00:002
1276977993425,7cyclictest37-21ksoftirqd/320:40:233
127696699341,31cyclictest0-21swapper/020:24:300
1276974993322,9cyclictest1281815-21sshd20:42:112
127697799325,6cyclictest131rcu_preempt19:22:093
1276977993020,6cyclictest1278967-21cron19:50:003
1276977993020,6cyclictest1278967-21cron19:50:003
1276977993016,11cyclictest1-21systemd19:55:003
1276977992813,10cyclictest37-21ksoftirqd/319:10:013
1276977992813,10cyclictest37-21ksoftirqd/319:10:003
1276974992812,13cyclictest1276976-21latency_hist19:10:012
1276974992812,13cyclictest1276976-21latency_hist19:10:002
127696699273,19cyclictest1277086-21wc19:10:020
127696699273,19cyclictest1277086-21wc19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional