You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-15 - 19:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sat Aug 15, 2026 12:43:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1256934995532,21cyclictest1256930-21cyclictest10:55:592
1256932995229,15cyclictest12650irq/32-eth007:11:000
125693499513,40cyclictest1403086-21ssh12:11:042
1256933995039,6cyclictest1282444-21kworker/u8:2-xprtiod10:00:011
1256934994939,6cyclictest1344716-21wc10:35:022
1256933994938,7cyclictest1393852-21needreboot11:55:571
1256932994941,5cyclictest151ktimers/010:01:030
1256932994941,5cyclictest151ktimers/010:01:030
125693499484,40cyclictest1348919-21ssh10:41:012
1256934994843,2cyclictest0-21swapper/209:31:052
1256933994839,6cyclictest1392787-21systemd-run11:55:011
1256933994838,4cyclictest0-21swapper/109:36:011
1256934994739,5cyclictest1335610-21awk10:20:012
1256933994730,12cyclictest1405876-21needreboot12:15:541
1256933994638,4cyclictest0-21swapper/111:36:001
1256933994637,4cyclictest0-21swapper/109:15:551
125693499452,16cyclictest1369863-21ssh11:16:022
1256932994525,14cyclictest12650irq/32-eth011:30:590
1256932994522,15cyclictest12650irq/32-eth010:56:000
1256934994434,6cyclictest1285427-21sh08:45:012
1256934994434,5cyclictest1290851-21kworker/u8:4-rpciod10:00:522
1256934994434,5cyclictest1290851-21kworker/u8:4-rpciod10:00:522
1256934994429,11cyclictest1345557-21kernelversion10:35:522
125693499441,20cyclictest1385035-21ssh11:41:032
1256932994411,15cyclictest14-21ksoftirqd/009:15:550
1256934994335,4cyclictest0-21swapper/209:46:042
125693499432,8cyclictest1309479-21ssh09:36:002
1256934994325,14cyclictest0-21swapper/211:31:102
1256934994320,19cyclictest0-21swapper/212:26:042
125693499431,4cyclictest1336781-21ssh10:20:582
125693499431,18cyclictest1363826-21ntpq11:05:552
1256932994336,4cyclictest1419917-21systemd-run12:40:020
1256932994335,6cyclictest421-21dbus-daemon07:20:010
125693499422,6cyclictest1351883-21ntpq10:45:592
125693499422,6cyclictest1277398-21ntpq08:16:022
1256934994218,21cyclictest1276412-21ls08:15:012
125693499421,14cyclictest1318531-21ntpq09:50:582
125693499421,11cyclictest1278783-21ntpq08:20:552
1256933994233,5cyclictest1003-21mta-sts-daemon10:01:001
1256933994233,5cyclictest1003-21mta-sts-daemon10:01:001
1256933994231,7cyclictest420-21avahi-daemon09:55:021
1256934994127,11cyclictest0-21swapper/212:40:012
125693499412,6cyclictest1393978-21ntpq11:56:022
125693499412,5cyclictest1375978-21ntpq11:26:002
125693499412,15cyclictest1300292-21ntpq09:20:592
125693499411,9cyclictest1265281-21ntpq07:35:562
125693499411,9cyclictest1263869-21ntpq07:30:592
125693499411,9cyclictest1263869-21ntpq07:30:592
125693499411,5cyclictest1287779-21ntpq08:50:562
125693499411,10cyclictest1286395-21ntpq08:46:042
1256933994132,6cyclictest1399559-21fschecks_count12:05:491
125693499403,9cyclictest1408982-21ssh12:20:572
125693499403,27cyclictest1321611-21rm09:55:592
1256934994032,5cyclictest1291445-21systemd-run09:05:012
1256934994029,7cyclictest265-21systemd-journal09:20:012
1256934994027,8cyclictest265-21systemd-journal12:15:132
125693499402,5cyclictest1266842-21ntpq07:40:572
125693499402,30cyclictest1400037-21ssh12:05:592
125693499402,15cyclictest1360836-21ntpq11:00:562
125693499402,14cyclictest1312475-21ntpq09:41:012
125693499401,5cyclictest1330732-21ntpq10:10:582
125693499401,5cyclictest1268411-21ntpq07:46:022
125693499401,11cyclictest1355812-21ssh10:52:162
125693499401,10cyclictest1258223-21timerwakeupswit07:11:072
1256933994031,6cyclictest1-21systemd11:05:011
1256933994030,6cyclictest459389-21cups-browsed12:29:051
1256933994029,7cyclictest459389-21cups-browsed09:28:161
1256932994024,14cyclictest1275994-21systemctl08:11:030
1256932994010,7cyclictest0-21swapper/009:56:000
125693499393,7cyclictest1294156-21ntpq09:11:002
125693499393,6cyclictest1260811-21ntpq07:20:572
125693499392,5cyclictest1281855-21ntpq08:30:592
125693499391,9cyclictest1269788-21ntpq07:50:572
125693499391,12cyclictest1327725-21ntpq10:05:592
125693499391,12cyclictest1327725-21ntpq10:05:592
125693499391,11cyclictest1387966-21ssh11:46:002
125693499391,10cyclictest1366585-21irqstats11:10:522
125693499391,10cyclictest1366585-21irqstats11:10:522
1256933993933,3cyclictest0-21swapper/112:34:141
1256933993932,3cyclictest0-21swapper/110:21:041
1256933993930,6cyclictest1384712-21latency11:40:531
1256932993934,3cyclictest1316034-21unixbench-2d09:46:100
1256932993933,4cyclictest1368697-21idleruntime-cro11:15:000
1256932993933,4cyclictest1368697-21idleruntime-cro11:15:000
1256932993932,4cyclictest1003-21mta-sts-daemon09:11:030
1256932993922,14cyclictest12650irq/32-eth012:10:570
125693299391,33cyclictest1342826-21ssh10:30:590
125693499382,7cyclictest1303308-21ssh09:26:002
125693499381,9cyclictest1381882-21ntpq11:35:572
125693499381,7cyclictest1258911-21/usr/sbin/munin07:15:492
1256934993817,18cyclictest459389-21cups-browsed08:09:592
125693499381,13cyclictest1292886-21users09:06:112
125693499381,12cyclictest1415980-21ssh12:32:122
1256933993826,9cyclictest1312473-21ntp_kernel_pll_09:41:021
1256933993826,9cyclictest1286111-21latency08:45:551
1256932993832,4cyclictest1291421-21cron09:05:010
1256932993831,4cyclictest1282492-21latency_hist08:35:020
1256932993824,7cyclictest171rcu_preempt11:35:560
1256932993824,11cyclictest1407816-21munin-run12:20:010
1256932993824,11cyclictest1386754-21cron11:45:010
1256934993730,3cyclictest0-21swapper/212:01:012
1256934993730,3cyclictest0-21swapper/212:01:012
125693499372,7cyclictest1272891-21ntpq08:01:002
125693499372,11cyclictest1339779-21ntpq10:25:592
125693499371,9cyclictest1271331-21ntpq07:55:582
125693499371,8cyclictest1262068-21irqstats07:25:512
125693499371,12cyclictest1390912-21ssh11:50:572
125693499371,11cyclictest1374659-21ssh11:24:522
1256933993724,10cyclictest1380927-21latency_hist11:35:021
1256932993731,4cyclictest1361059-21wc11:01:030
125693499361,9cyclictest1289949-21munin-run09:00:012
125693499361,9cyclictest1289949-21munin-run09:00:012
1256934993613,20cyclictest1276389-21kworker/u8:1-xprtiod08:35:502
1256933993630,3cyclictest0-21swapper/108:40:011
1256933993630,3cyclictest0-21swapper/107:40:001
1256933993629,3cyclictest0-21swapper/111:10:471
1256933993629,3cyclictest0-21swapper/111:10:471
1256933993628,5cyclictest1277913-21cron08:20:011
1256933993625,8cyclictest1292954-21rm09:10:021
1256933993622,9cyclictest0-21swapper/110:45:561
1256932993629,4cyclictest1292926-21diskmemload09:10:020
1256932993628,5cyclictest1352287-21timerwakeupswit10:46:080
1256932993627,6cyclictest1302304-21ssh09:25:040
125693299362,28cyclictest958-21sendmail-mta10:41:020
125693499351,8cyclictest1280301-21awk08:25:562
1256933993529,3cyclictest0-21swapper/109:25:011
1256933993528,5cyclictest0-21swapper/109:31:051
1256933993528,4cyclictest1314857-21aten_r2power_en09:45:471
1256933993527,5cyclictest1271909-21grep08:00:011
1256932993529,4cyclictest1314456-21latency_hist09:45:020
1256932993526,6cyclictest485-21in:imuxsock10:25:050
1256932993516,14cyclictest12650irq/32-eth009:51:030
125693499341,8cyclictest1256947-21munin-run07:10:012
1256933993427,4cyclictest1001-21mta-sts-daemon11:15:251
1256933993426,5cyclictest1398786-21ssh12:05:001
1256933993426,5cyclictest1398786-21ssh12:05:001
1256933993426,5cyclictest1294583-21timerwakeupswit09:11:081
1256933993426,5cyclictest1254388-21kworker/u8:5+xprtiod07:20:011
1256933993425,6cyclictest1363253-21aten_r2power_fr11:05:461
1256933993424,7cyclictest958-21sendmail-mta09:01:021
1256932993422,9cyclictest12650irq/32-eth007:50:560
1256932993422,2cyclictest171rcu_preempt08:57:050
1256932993422,2cyclictest171rcu_preempt08:57:050
1256932993420,3cyclictest171rcu_preempt08:35:570
125693399334,23cyclictest1342779-21ssh10:30:571
1256933993325,5cyclictest1373057-21processes11:21:051
1256933993325,5cyclictest1-21systemd10:41:041
1256933993324,6cyclictest1292926-21diskmemload12:36:091
1256933993320,10cyclictest1262276-21ntp_kernel_err07:25:551
1256932993326,5cyclictest0-21swapper/010:20:010
1256932993326,4cyclictest1271908-21munin-run08:00:010
1256932993325,5cyclictest459389-21cups-browsed07:30:010
1256932993324,6cyclictest1308243-21sh09:35:020
1256932993322,8cyclictest1388368-21timerandwakeup11:46:090
125693299332,24cyclictest1408999-21sh12:20:570
1256933993223,6cyclictest1288429-21systemd-run08:55:011
1256933993220,9cyclictest1409355-21switchtime12:21:081
125693399321,23cyclictest1330700-21ssh10:10:561
1256932993226,4cyclictest420-21avahi-daemon09:35:470
1256932993226,3cyclictest1264444-21kworker/u8:15-xprtiod07:50:010
1256932993219,2cyclictest171rcu_preempt07:41:010
1256933993125,3cyclictest0-21swapper/108:10:011
1256933993123,5cyclictest1282329-21unixbench_multi08:31:081
1256933993121,7cyclictest1340238-21timerandwakeup10:26:151
1256933993118,9cyclictest1357116-21/usr/sbin/munin10:55:541
125693299315,9cyclictest14-21ksoftirqd/010:11:000
125693299314,10cyclictest14-21ksoftirqd/011:26:000
1256932993125,3cyclictest0-21swapper/011:50:540
125693299312,23cyclictest1415041-21ssh12:30:590
1256932993118,2cyclictest171rcu_preempt08:20:570
1256932993117,3cyclictest171rcu_preempt12:01:340
1256932993117,3cyclictest171rcu_preempt12:01:340
1256932993117,3cyclictest171rcu_preempt11:05:530
125693299311,15cyclictest1369814-21ntpq11:15:590
1256932993111,3cyclictest171rcu_preempt09:20:560
125693399303,14cyclictest1354856-21ssh10:51:021
1256933993024,3cyclictest1255600-21kworker/u8:11-xprtiod07:10:011
1256933993021,6cyclictest1335600-21sh10:20:011
1256933993019,6cyclictest1280924-21latency_hist08:30:021
125693399301,23cyclictest1345788-21ssh10:35:571
125693299307,9cyclictest14-21ksoftirqd/010:20:580
125693299306,9cyclictest14-21ksoftirqd/010:51:050
125693299303,21cyclictest1399929-21ssh12:05:550
125693299302,20cyclictest1394051-21ssh11:56:050
1256932993015,3cyclictest171rcu_preempt10:39:400
1256932993012,4cyclictest171rcu_preempt12:27:200
1256932993010,3cyclictest171rcu_preempt11:21:100
1256933992922,3cyclictest0-21swapper/112:11:051
1256933992921,5cyclictest420-21avahi-daemon08:42:281
1256933992920,4cyclictest0-21swapper/111:28:301
125693299295,10cyclictest14-21ksoftirqd/007:30:560
125693299295,10cyclictest14-21ksoftirqd/007:30:560
1256932992922,3cyclictest0-21swapper/008:16:030
1256932992918,5cyclictest1000-21mta-sts-daemon10:05:050
1256932992918,5cyclictest1000-21mta-sts-daemon10:05:050
1256932992916,2cyclictest171rcu_preempt08:00:560
1256932992913,3cyclictest171rcu_preempt08:50:020
125693399283,19cyclictest459389-21cups-browsed07:45:591
1256933992818,7cyclictest0-21swapper/111:45:511
1256932992821,4cyclictest191rcuc/007:40:010
1256932992819,2cyclictest171rcu_preempt08:10:020
1256932992817,3cyclictest171rcu_preempt07:20:560
125693399273,18cyclictest1263899-21sshd07:31:011
125693399273,18cyclictest1263899-21sshd07:31:011
1256933992718,6cyclictest459367-21cupsd10:07:551
1256933992718,6cyclictest459367-21cupsd10:07:551
1256932992718,7cyclictest1003-21mta-sts-daemon08:40:480
1256932992716,9cyclictest0-21swapper/007:10:010
1256932992712,3cyclictest171rcu_preempt08:50:560
1256933992618,5cyclictest1261425-21cron07:25:011
1256933992615,8cyclictest1266775-21ntp_kernel_err07:40:551
125693299264,10cyclictest14-21ksoftirqd/008:25:560
1256933992519,3cyclictest0-21swapper/107:55:011
125693399251,19cyclictest1272905-21fake-hwclock08:01:011
125693399251,18cyclictest1257879-21ntpq07:11:001
125693399241,5cyclictest1289908-21munin-run09:00:011
125693399241,5cyclictest1289908-21munin-run09:00:011
1256933992312,8cyclictest1275762-21ntp_kernel_err08:10:551
1256933992312,8cyclictest1002-21mta-sts-daemon08:21:021
125693599195,6cyclictest0-21swapper/311:46:003
125693599192,8cyclictest0-21swapper/309:10:583
125693599183,8cyclictest0-21swapper/312:25:573
125693599183,8cyclictest0-21swapper/308:01:013
125693599183,6cyclictest0-21swapper/309:20:563
125693599182,8cyclictest0-21swapper/312:35:593
125693599182,8cyclictest0-21swapper/310:40:553
125693599182,8cyclictest0-21swapper/309:15:593
125693599182,7cyclictest0-21swapper/312:10:583
125693599173,8cyclictest0-21swapper/310:30:573
125693599173,7cyclictest0-21swapper/309:41:113
125693599172,8cyclictest0-21swapper/312:30:573
125693599172,8cyclictest0-21swapper/310:55:593
125693599172,6cyclictest0-21swapper/308:50:563
125693599166,7cyclictest0-21swapper/310:00:503
125693599166,7cyclictest0-21swapper/310:00:503
125693599163,7cyclictest0-21swapper/311:21:013
125693599163,7cyclictest0-21swapper/310:26:103
125693599163,7cyclictest0-21swapper/310:10:573
125693599163,7cyclictest0-21swapper/309:50:593
125693599163,6cyclictest0-21swapper/309:30:563
125693599162,8cyclictest0-21swapper/311:16:033
125693599162,8cyclictest0-21swapper/310:46:023
125693599162,7cyclictest0-21swapper/312:00:563
125693599162,7cyclictest0-21swapper/312:00:563
125693599162,7cyclictest0-21swapper/311:31:023
125693599162,7cyclictest0-21swapper/310:51:013
125693599162,7cyclictest0-21swapper/310:16:093
125693599162,7cyclictest0-21swapper/309:46:043
125693599161,7cyclictest0-21swapper/310:20:563
125693599161,7cyclictest0-21swapper/309:25:573
125693599161,6cyclictest0-21swapper/308:55:563
125693599161,6cyclictest0-21swapper/308:55:563
125693599161,6cyclictest0-21swapper/307:21:013
125693599153,7cyclictest0-21swapper/311:40:563
125693599153,7cyclictest0-21swapper/309:55:573
125693599153,6cyclictest0-21swapper/307:11:003
125693599152,7cyclictest0-21swapper/312:06:003
125693599152,6cyclictest0-21swapper/311:25:593
125693599152,6cyclictest0-21swapper/308:35:573
125693599151,8cyclictest0-21swapper/309:35:583
125693599151,7cyclictest0-21swapper/312:20:583
125693599151,7cyclictest0-21swapper/311:51:043
125693599151,7cyclictest0-21swapper/311:11:003
125693599151,7cyclictest0-21swapper/311:11:003
125693599151,7cyclictest0-21swapper/311:00:553
125693599151,7cyclictest0-21swapper/310:05:583
125693599151,7cyclictest0-21swapper/310:05:583
125693599143,4cyclictest0-21swapper/307:25:553
125693599142,7cyclictest0-21swapper/311:55:593
125693599142,7cyclictest0-21swapper/311:05:543
125693599141,7cyclictest0-21swapper/309:06:013
125693599141,6cyclictest0-21swapper/307:35:573
125693599136,5cyclictest0-21swapper/308:31:003
125693599132,7cyclictest0-21swapper/310:35:473
125693599132,6cyclictest0-21swapper/308:46:013
125693599132,6cyclictest0-21swapper/308:26:033
125693599131,7cyclictest0-21swapper/312:16:013
125693599131,6cyclictest0-21swapper/308:15:563
125693599131,6cyclictest0-21swapper/307:40:573
125693599122,6cyclictest0-21swapper/311:35:513
125693599121,6cyclictest0-21swapper/308:05:573
125693599121,6cyclictest0-21swapper/307:55:563
125693599121,6cyclictest0-21swapper/307:50:593
125693599113,4cyclictest0-21swapper/308:10:563
125693599112,5cyclictest0-21swapper/308:21:113
125693599111,7cyclictest0-21swapper/307:10:023
125693599105,3cyclictest0-21swapper/309:01:123
125693599103,4cyclictest0-21swapper/308:40:573
12569359994,3cyclictest0-21swapper/307:45:373
12569359994,3cyclictest0-21swapper/307:30:573
12569359994,3cyclictest0-21swapper/307:30:573
12569359991,5cyclictest0-21swapper/307:16:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional