You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 19:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jun 08, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28573521280,3sleep33056-21PK-Backend11:35:133
1902842850,5sleep23056-21PK-Backend07:05:122
1902842850,5sleep23056-21PK-Backend07:05:122
19132428262,14sleep10-21swapper/107:06:581
19132428262,14sleep10-21swapper/107:06:571
19011527745,22sleep00-21swapper/007:05:120
19011527745,22sleep00-21swapper/007:05:110
19123927438,13sleep30-21swapper/307:05:523
19123927438,13sleep30-21swapper/307:05:513
2228582560,4sleep33056-21PK-Backend08:35:173
2824372540,4sleep23056-21PK-Backend11:25:142
1929662480,4sleep10-21swapper/107:10:221
2657912430,4sleep33056-21PK-Backend10:40:003
19173499383,22cyclictest253666-21perf10:05:002
19172999358,18cyclictest207566-21taskset07:51:471
19172999308,9cyclictest274186-21runrttasks11:01:331
19172999307,11cyclictest237473-21runrttasks09:19:141
19172999307,10cyclictest278049-21taskset11:13:211
19172999297,8cyclictest295335-21runrttasks12:02:441
19172999297,8cyclictest265754-21taskset10:39:311
19172999297,8cyclictest262212-21taskset10:28:331
19172999287,8cyclictest251761-21taskset09:58:011
19172999284,15cyclictest0-21swapper/110:10:101
2634082270,6sleep234-21ksoftirqd/210:30:202
191734992710,15cyclictest271275-21sshd10:55:122
19172999278,17cyclictest201944-21processes07:35:191
19172999278,17cyclictest201944-21processes07:35:191
19172999277,8cyclictest302560-21taskset12:23:091
19172999276,8cyclictest255262-21taskset10:08:281
191725992716,7cyclictest436-20systemd-journal07:15:010
191729992618,6cyclictest200841-21perf07:34:591
19173699258,5cyclictest306581-21sed12:35:003
19173699250,23cyclictest3056-21PK-Backend09:30:153
19173499259,14cyclictest978-21snmpd09:24:102
19173499259,14cyclictest303431-21awk12:25:152
191734992510,13cyclictest283816-21cpuspeed_turbos11:30:122
19173499250,19cyclictest0-21swapper/207:27:052
191725992517,5cyclictest255408-21basename10:10:000
191725992511,10cyclictest436-20systemd-journal07:40:000
191725992511,10cyclictest436-20systemd-journal07:39:590
191725992510,10cyclictest436-20systemd-journal12:00:010
139099250,11rtkit-daemon3056-21PK-Backend10:22:433
2583522240,9sleep234-21ksoftirqd/210:15:202
19173499249,13cyclictest286855-21sh11:40:002
191734992414,9cyclictest213295-21apt08:10:112
19172999247,6cyclictest249665-21taskset09:51:401
19172999247,3cyclictest221904-21taskset08:34:331
19172999244,8cyclictest0-21swapper/111:20:131
19172599241,4cyclictest260594-21cron10:24:590
2964752230,2chrt0-21swapper/112:05:191
191736992321,1cyclictest3056-21PK-Backend08:13:593
191736992311,10cyclictest211539-21sshd08:05:123
19173499239,12cyclictest266662-21grep10:40:182
19173499239,12cyclictest218565-21grep08:25:002
19173499237,3cyclictest277710-21timerandwakeup11:10:202
19173499236,4cyclictest978-21snmpd10:12:122
19173499236,15cyclictest0-21swapper/212:05:002
19173499235,6cyclictest250851-21cat09:55:152
19173499235,3cyclictest0-21swapper/211:53:212
191734992310,11cyclictest275385-21tail11:05:192
19173499230,5cyclictest978-21snmpd11:43:272
19173499230,21cyclictest297394-21cat12:10:022
19172999239,12cyclictest209593-21unin-run08:00:001
19172999234,7cyclictest0-21swapper/112:20:011
19172999231,3cyclictest232121-21runrttasks09:02:141
191729992312,9cyclictest286260-21python311:35:211
191729992312,8cyclictest242147-21python309:30:191
191729992311,10cyclictest243017-21apt09:35:121
2991042220,2sleep0161rcuc/012:15:000
2991042220,2sleep0161rcuc/012:15:000
191736992220,1cyclictest3056-21PK-Backend09:53:563
191736992219,2cyclictest3056-21PK-Backend10:14:043
191736992219,2cyclictest3056-21PK-Backend08:22:373
19173699220,20cyclictest3056-21PK-Backend09:56:343
19173499227,13cyclictest223024-21processes08:35:202
19173499224,6cyclictest0-21swapper/211:20:242
19173499221,7cyclictest254391-21if_err_enp2s010:05:172
19173499221,7cyclictest204373-21turbostat07:45:002
191734992216,4cyclictest234083-21perf09:10:012
191734992210,10cyclictest1095-21runrttasks08:59:152
19173499220,9cyclictest235199-21grep09:10:212
19173499220,8cyclictest248916-21awk09:50:142
19173499220,7cyclictest245417-21grep09:40:172
19173499220,7cyclictest237304-21chrt09:17:022
19173499220,19cyclictest0-21swapper/209:47:112
191729992215,5cyclictest0-21swapper/112:10:211
191729992215,5cyclictest0-21swapper/112:10:201
191729992213,7cyclictest218517-21unin-run08:25:001
191729992213,7cyclictest1-21systemd10:35:001
191729992213,6cyclictest292329-21apt11:55:121
191729992212,8cyclictest282559-21sed11:25:141
191729992212,8cyclictest266049-21/usr/sbin/munin10:40:231
191729992212,8cyclictest247966-21idleruntime-cro09:50:011
191729992212,8cyclictest245004-21cstates09:40:111
191729992212,8cyclictest227784-21/usr/sbin/munin08:50:151
191729992212,8cyclictest226627-21threads08:45:211
191729992212,8cyclictest199142-21unin-run07:30:001
191729992211,9cyclictest234839-21kernelversion09:10:161
191729992211,9cyclictest230721-21date09:00:001
191729992211,9cyclictest216767-21turbostat08:20:011
191729992211,8cyclictest233355-21ntp_states09:05:181
191725992220,1cyclictest0-21swapper/010:27:140
191725992215,4cyclictest1895007-21snapd12:20:520
191725992213,5cyclictest200855-21munin-plugin-st07:35:000
191725992212,8cyclictest883-21dbus-daemon10:15:000
191725992212,7cyclictest209567-21latency_hist08:00:010
19172599220,21cyclictest203560-21needreboot07:40:170
19172599220,21cyclictest0-21swapper/008:27:430
139099220,16rtkit-daemon0-21swapper/311:25:583
139099220,15rtkit-daemon3056-21PK-Backend11:56:573
139099220,15rtkit-daemon3056-21PK-Backend11:11:113
191736992119,1cyclictest3056-21PK-Backend12:10:173
191736992119,1cyclictest3056-21PK-Backend12:10:173
191736992119,1cyclictest3056-21PK-Backend11:33:403
191736992119,1cyclictest3056-21PK-Backend10:49:533
191736992118,2cyclictest3056-21PK-Backend10:35:003
191736992118,2cyclictest3056-21PK-Backend08:32:183
191736992118,2cyclictest3056-21PK-Backend07:24:103
191736992113,6cyclictest273864-21sensors_temp11:00:223
19173699210,20cyclictest3056-21PK-Backend12:03:553
19173699210,19cyclictest3056-21PK-Backend09:45:173
19173699210,19cyclictest3056-21PK-Backend08:25:153
19173699210,19cyclictest3056-21PK-Backend07:26:473
19173499219,11cyclictest3056-21PK-Backend07:20:302
19173499218,11cyclictest978-21snmpd09:36:392
19173499217,3cyclictest34-21ksoftirqd/208:27:142
19173499217,12cyclictest271006-21latency_hist10:55:002
19173499215,3cyclictest221006-21latency08:30:152
19173499215,3cyclictest194356-21tr07:15:122
19173499211,8cyclictest207830-21/usr/sbin/munin07:55:002
191734992113,6cyclictest213015-21cron08:10:012
19173499210,8cyclictest264494-21cpuspeed_turbos10:35:142
19173499210,8cyclictest224706-21seq08:40:202
19173499210,7cyclictest293273-21timerandwakeup11:55:222
19173499210,7cyclictest261602-21awk10:25:182
19173499210,7cyclictest240130-21/usr/sbin/munin09:25:162
19173499210,7cyclictest193191-21chrt07:12:152
19172999216,13cyclictest26-21ksoftirqd/110:50:141
191729992113,6cyclictest436-20systemd-journal11:30:121
191729992113,6cyclictest224700-21ntp_states08:40:201
191729992113,6cyclictest193896-21cat07:15:021
191729992113,5cyclictest222225-21apt08:35:111
191729992113,5cyclictest195916-21apt07:20:121
191729992112,7cyclictest307617-21sed12:35:211
191729992112,7cyclictest290575-21/usr/sbin/munin11:50:101
191729992112,7cyclictest259351-21/usr/sbin/munin10:20:121
191729992112,7cyclictest213482-21cpuspeed_turbos08:10:111
191729992112,7cyclictest203551-21grep07:40:171
191729992112,5cyclictest257407-21apt10:15:121
191729992111,8cyclictest303354-21df_inode12:25:131
191729992111,8cyclictest289342-21/usr/sbin/munin11:45:171
191729992111,8cyclictest280139-21sed11:20:001
191729992111,8cyclictest271852-21chrt10:55:171
191729992111,8cyclictest238850-21/usr/sbin/munin09:20:221
191729992111,8cyclictest209869-21cpu08:00:101
191729992110,9cyclictest287533-21smartctl11:40:161
191729992110,9cyclictest213211-21cat08:10:011
19172999210,20cyclictest0-21swapper/111:09:501
19172999210,20cyclictest0-21swapper/109:29:181
19172599212,13cyclictest294061-21apt12:00:110
191725992119,1cyclictest0-21swapper/011:47:510
191725992119,1cyclictest0-21swapper/011:35:270
191725992119,1cyclictest0-21swapper/010:57:150
191725992119,1cyclictest0-21swapper/010:53:030
191725992119,1cyclictest0-21swapper/009:58:490
191725992119,1cyclictest0-21swapper/009:54:230
191725992119,1cyclictest0-21swapper/009:15:490
191725992119,1cyclictest0-21swapper/008:11:130
191725992119,1cyclictest0-21swapper/007:48:490
191725992119,1cyclictest0-21swapper/007:26:320
191725992119,1cyclictest0-21swapper/007:18:200
191725992116,3cyclictest227570-21cpuspeed_turbos08:50:140
191725992115,5cyclictest13-21ksoftirqd/011:30:110
191725992113,6cyclictest288437-21sshd11:43:120
191725992112,6cyclictest258864-21cron10:20:000
191725992110,8cyclictest883-21dbus-daemon09:05:010
19172599210,20cyclictest0-21swapper/012:25:500
19172599210,20cyclictest0-21swapper/008:42:020
19172599210,19cyclictest0-21swapper/008:35:470
139099210,4rtkit-daemon3056-21PK-Backend11:23:383
139099210,14rtkit-daemon3056-21PK-Backend10:42:553
191736992018,1cyclictest3056-21PK-Backend12:27:213
191736992018,1cyclictest3056-21PK-Backend10:29:453
191736992018,1cyclictest3056-21PK-Backend10:15:533
191736992018,1cyclictest3056-21PK-Backend10:04:253
191736992018,1cyclictest3056-21PK-Backend09:39:023
191736992018,1cyclictest3056-21PK-Backend09:29:103
191736992018,1cyclictest3056-21PK-Backend09:24:423
191736992018,1cyclictest3056-21PK-Backend09:17:393
191736992018,1cyclictest3056-21PK-Backend09:12:263
191736992018,1cyclictest3056-21PK-Backend09:05:233
191736992018,1cyclictest3056-21PK-Backend07:50:403
191736992018,1cyclictest3056-21PK-Backend07:40:193
191736992017,2cyclictest3056-21PK-Backend08:03:113
19173699200,1cyclictest3056-21PK-Backend11:46:513
19173699200,1cyclictest3056-21PK-Backend11:07:303
19173699200,1cyclictest3056-21PK-Backend10:55:083
19173699200,19cyclictest3056-21PK-Backend09:40:513
19173699200,19cyclictest3056-21PK-Backend07:46:483
19173699200,19cyclictest3056-21PK-Backend07:16:263
19173699200,18cyclictest3056-21PK-Backend12:06:183
19173699200,18cyclictest3056-21PK-Backend08:50:493
19173499207,3cyclictest259741-21missed_timers10:20:152
19173499206,12cyclictest0-21swapper/207:45:112
19173499205,4cyclictest299969-21seq12:15:162
19173499205,3cyclictest0-21swapper/212:38:192
19173499204,14cyclictest0-21swapper/212:30:022
191734992018,1cyclictest0-21swapper/211:03:362
191734992018,1cyclictest0-21swapper/210:48:082
191734992018,1cyclictest0-21swapper/208:45:542
191734992018,1cyclictest0-21swapper/208:16:142
191734992018,1cyclictest0-21swapper/207:39:032
191734992018,1cyclictest0-21swapper/207:39:022
191734992014,5cyclictest0-21swapper/207:56:322
191734992014,4cyclictest279920-21sh11:19:592
19173499200,7cyclictest227535-21cpuspeed_turbos08:50:132
19173499200,19cyclictest0-21swapper/212:10:552
19173499200,19cyclictest0-21swapper/212:10:552
19173499200,19cyclictest0-21swapper/211:45:272
19173499200,19cyclictest0-21swapper/209:00:412
19173499200,19cyclictest0-21swapper/207:30:442
191729992012,7cyclictest268171-21/usr/sbin/munin10:45:131
191729992012,6cyclictest252390-21cpuspeed_turbos10:00:111
191729992011,7cyclictest305251-21cat12:30:121
191729992011,7cyclictest205376-21ntp_states07:45:191
191729992010,8cyclictest219084-21expr08:25:121
191725992018,1cyclictest0-21swapper/011:01:270
191725992018,1cyclictest0-21swapper/010:42:210
191725992018,1cyclictest0-21swapper/009:22:040
191725992018,1cyclictest0-21swapper/008:49:190
191725992018,1cyclictest0-21swapper/008:03:370
191725992017,2cyclictest0-21swapper/009:40:150
191725992017,2cyclictest0-21swapper/008:57:380
19172599200,1cyclictest0-21swapper/010:36:060
19172599200,1cyclictest0-21swapper/009:26:310
19172599200,19cyclictest0-21swapper/011:25:270
19172599200,19cyclictest0-21swapper/011:05:400
19172599200,19cyclictest0-21swapper/010:34:160
19172599200,19cyclictest0-21swapper/009:49:550
19172599200,19cyclictest0-21swapper/007:50:530
191736991917,1cyclictest3056-21PK-Backend12:36:263
191736991917,1cyclictest3056-21PK-Backend12:23:353
191736991917,1cyclictest3056-21PK-Backend08:46:233
191736991917,1cyclictest3056-21PK-Backend08:41:573
19173699190,1cyclictest3056-21PK-Backend11:53:003
19173699190,1cyclictest3056-21PK-Backend11:40:153
19173699190,1cyclictest3056-21PK-Backend10:09:383
19173699190,1cyclictest3056-21PK-Backend09:04:223
19173699190,1cyclictest3056-21PK-Backend08:58:543
19173699190,1cyclictest3056-21PK-Backend07:56:223
19173699190,1cyclictest3056-21PK-Backend07:39:193
19173699190,1cyclictest3056-21PK-Backend07:39:193
19173699190,1cyclictest3056-21PK-Backend07:33:503
19173699190,18cyclictest3056-21PK-Backend10:51:423
19172599192,12cyclictest267807-21apt10:45:110
191725991917,1cyclictest0-21swapper/011:50:150
191725991917,1cyclictest0-21swapper/009:11:220
191725991917,1cyclictest0-21swapper/008:31:210
191725991917,1cyclictest0-21swapper/008:07:480
191725991914,3cyclictest278920-21if_err_enp2s011:15:140
191725991912,5cyclictest2156-21gnome-shell07:24:080
19172599190,1cyclictest0-21swapper/012:09:070
19172599190,1cyclictest0-21swapper/011:10:400
19172599190,1cyclictest0-21swapper/009:31:580
19172599190,1cyclictest0-21swapper/009:05:550
191736991816,1cyclictest0-21swapper/308:15:483
19173499181,6cyclictest241695-21idleruntime09:30:142
191734991816,1cyclictest0-21swapper/208:02:342
19172599189,7cyclictest244466-21cron09:40:000
191725991816,1cyclictest0-21swapper/008:22:290
191725991812,4cyclictest299983-21cat12:15:160
191725991811,5cyclictest305868-21awk12:30:210
191725991811,4cyclictest253671-21idleruntime-cro10:05:000
19173499170,5cyclictest301412-21tr12:20:142
19172599179,5cyclictest1095-21runrttasks08:15:200
19172599169,5cyclictest307029-21ls12:35:120
19172599161,12cyclictest280202-21apt11:20:140
191736991412,1cyclictest3056-21PK-Backend12:15:033
191736991410,3cyclictest3056-21PK-Backend11:15:323
19173699139,3cyclictest3056-21PK-Backend07:13:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional