You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 23:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Sep 19, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1369829279,9sleep00-21swapper/007:08:590
1367628471,9sleep30-21swapper/307:08:413
1365328370,9sleep20-21swapper/207:08:242
1359228164,13sleep10-21swapper/107:07:341
1386299250,24cyclictest0-21swapper/009:20:410
13870992220,2cyclictest11250-21kworker/3:010:22:333
13864992219,3cyclictest0-21swapper/112:11:301
1386299218,7cyclictest7659-21apt-get08:05:000
1387099202,18cyclictest0-21swapper/311:56:193
1386899200,18cyclictest0-21swapper/210:17:552
13864992018,2cyclictest2359-21kworker/1:111:02:131
13864992010,6cyclictest31111-21sh10:21:501
1386499200,1cyclictest0-21swapper/111:11:201
1387099193,15cyclictest0-21swapper/310:38:363
13870991918,1cyclictest2862-21kworker/3:112:11:423
13870991917,1cyclictest11250-21kworker/3:010:15:023
1387099190,18cyclictest2862-21kworker/3:112:21:423
13862991911,3cyclictest32617-21sh12:00:150
1386499186,10cyclictest25208-21ssh12:15:141
1386499182,15cyclictest0-21swapper/111:50:151
13864991817,1cyclictest0-21swapper/109:34:021
1386499180,9cyclictest0-21swapper/112:32:351
1386299187,6cyclictest31115-21idleruntime-cro11:40:000
1386299181,15cyclictest12773-21/usr/sbin/munin10:50:170
74352170,0sleep10-21swapper/109:49:071
13870991716,1cyclictest2862-21kworker/3:111:11:023
13870991715,1cyclictest0-21swapper/308:15:133
1387099171,1cyclictest0-21swapper/311:48:283
1387099171,15cyclictest1142-21runrttasks09:40:183
1386499170,15cyclictest25522-21sensors09:20:251
1386299175,6cyclictest6049-21munin-run11:25:010
286732160,0sleep10-21swapper/112:35:361
13870991615,1cyclictest2862-21kworker/3:112:28:533
13870991615,1cyclictest2862-21kworker/3:111:44:433
13870991615,1cyclictest2862-21kworker/3:111:36:333
1387099160,14cyclictest0-21swapper/310:12:123
1386899160,2cyclictest0-21swapper/211:37:132
13864991615,1cyclictest0-21swapper/109:52:291
1386499160,16cyclictest0-21swapper/109:36:361
1386299161,8cyclictest32061-21tr12:00:010
1386299160,16cyclictest0-21swapper/009:56:590
1387099155,8cyclictest4495-21apt-get10:45:143
13870991514,1cyclictest2862-21kworker/3:111:30:533
13870991514,1cyclictest2862-21kworker/3:111:30:523
13870991514,1cyclictest11250-21kworker/3:009:25:423
1387099150,0cyclictest0-21swapper/310:56:553
1386899151,2cyclictest0-21swapper/207:25:142
1386899150,0cyclictest0-21swapper/211:03:342
1386499158,3cyclictest6668-21diskmemload11:40:311
1386299152,2cyclictest141rcu_preempt12:15:310
1386299151,7cyclictest21617-21basename11:15:000
13862991512,2cyclictest19126-21sh11:51:250
1386299150,14cyclictest0-21swapper/010:17:490
78072140,1sleep20-21swapper/209:49:252
1387099141,12cyclictest0-21swapper/309:54:593
1386499146,6cyclictest0-21swapper/110:00:161
13864991410,1cyclictest0-21swapper/111:06:031
1386499140,13cyclictest0-21swapper/110:51:401
1386299147,4cyclictest0-21swapper/009:50:030
1386299142,11cyclictest19532-21/usr/sbin/munin12:30:220
13862991413,1cyclictest0-21swapper/009:35:430
13862991410,2cyclictest4658-21ssh09:46:560
1387099134,7cyclictest15590-21apt-get09:15:163
13870991311,1cyclictest0-21swapper/310:26:173
13870991310,2cyclictest465-21cron07:50:003
13870991310,2cyclictest27137-21forks12:35:173
13870991310,2cyclictest20526-21grep08:30:153
13870991310,2cyclictest10324-21grep12:05:253
13870991310,1cyclictest6668-21diskmemload11:50:053
1387099130,11cyclictest15015-21munin-run09:14:593
1386899139,3cyclictest6068-21wc11:25:012
13868991313,0cyclictest0-21swapper/211:42:332
1386899131,11cyclictest0-21swapper/212:13:342
13868991310,2cyclictest9048-21grep09:10:252
13868991310,2cyclictest5648-21grep09:05:202
13868991310,2cyclictest30897-21grep07:45:162
13868991310,2cyclictest26035-21grep12:15:172
13868991310,2cyclictest25781-21grep09:20:282
13868991310,2cyclictest22246-21grep10:35:282
13868991310,2cyclictest2158-21grep12:00:302
13868991310,2cyclictest19262-21cat12:30:202
13868991310,2cyclictest16616-21cut09:35:152
13868991310,2cyclictest15996-21wc09:35:002
1386899130,2cyclictest0-21swapper/211:32:222
1386899130,2cyclictest0-21swapper/211:32:212
1386899130,0cyclictest0-21swapper/211:29:442
1386499137,5cyclictest24432-21cut07:30:161
1386499135,4cyclictest0-21swapper/109:55:151
1386499131,8cyclictest0-21swapper/112:20:131
13864991312,1cyclictest0-21swapper/110:09:241
13864991312,1cyclictest0-21swapper/107:20:161
1386499131,11cyclictest6668-21diskmemload09:26:151
13864991311,1cyclictest18142-21apache211:56:081
13864991311,1cyclictest0-21swapper/108:15:131
1386499130,12cyclictest27179-21sh09:40:421
1386499130,12cyclictest0-21swapper/110:35:161
1386499130,11cyclictest0-21swapper/111:38:151
1386299139,3cyclictest15912-21ssh09:35:000
1386299131,1cyclictest26584-21apt-get10:20:020
13862991310,2cyclictest2880-21cron07:55:000
13862991310,2cyclictest1454-21munin-plugin-st10:05:000
13862991310,2cyclictest142022sleep011:49:060
13862991310,2cyclictest11402-21grep12:25:260
1387099129,2cyclictest6171-21grep08:00:163
1387099129,2cyclictest28011-21grep10:00:283
1387099129,2cyclictest26693-21grep08:40:293
1387099129,2cyclictest21588-21grep07:25:153
1387099129,2cyclictest20531-21sh10:54:543
1387099129,2cyclictest19407-21ssh09:55:283
1387099129,2cyclictest14363-21basename11:30:003
1387099129,2cyclictest13515-21cat10:30:263
1387099129,1cyclictest0-21swapper/309:32:303
1387099128,3cyclictest18183-21sed07:15:253
1387099126,5cyclictest6668-21diskmemload11:17:173
1387099126,5cyclictest2276-21sh11:02:563
1387099124,6cyclictest6668-21diskmemload11:06:523
13870991211,1cyclictest0-21swapper/309:22:393
13870991210,1cyclictest17284-21grep09:35:173
13870991210,1cyclictest1494-21head08:55:283
1387099120,3cyclictest0-21swapper/307:20:143
1386899129,2cyclictest9233-21grep08:05:192
1386899129,2cyclictest7744-21ssh11:45:142
1386899129,2cyclictest7660-21ls08:05:002
1386899129,2cyclictest4387-21tr07:55:212
1386899129,2cyclictest31925-21cron08:55:002
1386899129,2cyclictest27997-21df08:45:172
1386899129,2cyclictest26325-21grep07:35:152
1386899129,2cyclictest22631-21grep10:55:242
1386899129,2cyclictest19189-21grep07:20:152
1386899129,2cyclictest16291-21head08:20:192
1386899129,2cyclictest16198-21cron07:15:002
1386899129,2cyclictest13639-21sed08:15:162
1386899129,2cyclictest1300-21grep09:25:222
1386899129,2cyclictest12410-21grep10:30:172
1386899129,2cyclictest11739-21wc10:10:232
1386899128,3cyclictest25804-21sleep07:34:592
1386899126,4cyclictest32-21ksoftirqd/210:50:142
1386899123,4cyclictest0-21swapper/211:05:152
1386899121,3cyclictest0-21swapper/211:15:172
13868991210,2cyclictest13853-21cyclictest10:06:232
1386899120,11cyclictest0-21swapper/211:55:562
1386499129,2cyclictest9762-21ssh12:05:211
1386499129,2cyclictest6854-21tr11:25:161
1386499129,2cyclictest38150irq/131-snd_hda09:15:201
1386499129,2cyclictest23601-21wc07:30:001
1386499129,2cyclictest20056-21tr10:15:221
1386499127,4cyclictest7587-21ssh10:46:181
1386499126,2cyclictest225-21systemd-journal10:34:591
13864991211,1cyclictest0-21swapper/112:26:321
1386499120,0cyclictest0-21swapper/110:41:491
1386299129,2cyclictest7582-21cron09:29:590
1386299129,2cyclictest6977-21cron11:45:010
1386299129,2cyclictest6761-21grep11:25:150
1386299129,2cyclictest26717-21grep08:40:280
1386299129,2cyclictest23356-21grep11:15:200
1386299129,2cyclictest22792-21basename08:35:150
1386299129,2cyclictest20606-21sed07:20:250
1386299129,2cyclictest18664-21run-parts07:20:000
1386299129,2cyclictest18221-21tr08:25:150
1386299129,2cyclictest18043-21grep12:10:200
1386299129,2cyclictest12290-21grep10:10:280
1386299128,3cyclictest9975-21cat10:10:000
1386299128,3cyclictest30428-21sh10:40:280
1386299124,4cyclictest225-21systemd-journal07:40:000
1386299123,6cyclictest0-21swapper/007:25:140
1386299120,2cyclictest0-21swapper/011:05:080
1387099119,1cyclictest3870-21head07:55:173
1387099119,1cyclictest19021-21grep08:25:193
1387099119,1cyclictest18066-21ssh12:30:143
1387099118,2cyclictest8398-21cut08:05:143
1387099118,2cyclictest762-21ssh09:45:133
1387099118,2cyclictest700-21expr12:00:183
1387099118,2cyclictest6320-21chrt09:05:493
1387099118,2cyclictest3547-21ssh10:44:403
1387099118,2cyclictest31061-21ssh12:18:293
1387099118,2cyclictest28145-21tune2fs08:45:193
1387099118,2cyclictest1798-21ssh10:05:073
1387099118,2cyclictest17110-21grep08:20:283
1387099118,2cyclictest16224-21awk07:15:003
1387099118,2cyclictest11772-21wc08:10:233
1387099114,6cyclictest537-21snmpd11:20:233
1386899119,1cyclictest29709-21grep07:40:212
1386899118,2cyclictest7306-21ssh10:27:012
1386899118,2cyclictest534-21sed08:55:192
1386899118,2cyclictest4470-21wc09:05:002
1386899118,2cyclictest31428-21ssh10:22:052
1386899118,2cyclictest30711-21ssh10:02:322
1386899118,2cyclictest29476-21ssh12:36:142
1386899118,2cyclictest25000-21cut09:40:142
1386899118,2cyclictest17739-21ssh11:12:032
1386899118,2cyclictest16847-21cat07:15:142
1386899118,2cyclictest1632-21expr12:20:142
1386899118,2cyclictest15693-21cut09:15:172
1386899118,2cyclictest10063-21sh12:05:242
1386899117,3cyclictest29470-21ssh10:40:192
1386899117,2cyclictest9584-21cut09:50:132
1386899115,2cyclictest225-21systemd-journal07:55:002
1386899112,4cyclictest18519-21tune2fs09:55:192
1386899110,10cyclictest25272-21wc08:40:172
1386499119,2cyclictest31496-21cut11:20:181
1386499119,1cyclictest4575-21ssh10:25:211
1386499119,1cyclictest3850-21sed09:00:251
1386499119,1cyclictest15724-21grep11:30:201
1386499119,1cyclictest15724-21grep11:30:201
1386499118,2cyclictest7566-21cut11:45:131
1386499118,2cyclictest6285-21tail08:00:161
1386499118,2cyclictest4288-21awk07:55:201
1386499118,2cyclictest3632-21sh12:01:361
1386499118,2cyclictest32274-21awk08:55:161
1386499118,2cyclictest2916-21grep07:55:001
1386499118,2cyclictest27718-21grep07:35:261
1386499118,2cyclictest21073-21grep08:30:171
1386499118,2cyclictest16195-21cron07:15:001
1386499118,2cyclictest12901-21ssh10:10:561
1386499118,2cyclictest11463-21sh09:12:141
1386499116,4cyclictest22439-21ssh10:55:221
1386499110,9cyclictest0-21swapper/108:35:141
1386499110,10cyclictest26276-21ssh11:17:091
1386499110,10cyclictest15631-21cut08:20:171
1386299118,2cyclictest7264-21ssh12:23:400
1386299118,2cyclictest7029-21ssh10:45:520
1386299118,2cyclictest6238-21ssh10:26:120
1386299118,2cyclictest3907-21expr07:55:170
1386299118,2cyclictest31924-21cron08:55:000
1386299118,2cyclictest31660-21basename07:45:180
1386299118,2cyclictest29686-21ssh12:36:240
1386299118,2cyclictest28759-21rt-features08:45:240
1386299118,2cyclictest28458-21sh09:41:420
1386299118,2cyclictest27411-21ssh10:39:310
1386299118,2cyclictest16082-21expr09:15:170
1386299118,2cyclictest14346-21cut07:10:150
1386299118,2cyclictest1375-21ssh11:02:140
1386299118,2cyclictest13606-21grep08:15:170
1386299118,2cyclictest13435-21ssh09:13:470
1386299118,2cyclictest12267-21ssh10:30:160
1386299117,3cyclictest22078-21grep10:55:180
1386299117,3cyclictest1974-21cat09:00:000
1386299113,4cyclictest14874-21/usr/sbin/munin11:30:260
1386299113,4cyclictest14874-21/usr/sbin/munin11:30:250
1386299113,3cyclictest225-21systemd-journal08:09:550
1386299110,1cyclictest0-21swapper/012:05:490
1387099108,2cyclictest2807-21chrt07:54:093
1387099108,2cyclictest24747-21sed07:30:183
1387099108,1cyclictest30079-21grep08:50:143
1387099108,1cyclictest26467-21expr07:35:153
1387099101,4cyclictest30267-21sshd07:41:233
1386899108,1cyclictest23497-21basename08:35:172
1386899107,2cyclictest9900-21expr12:25:162
1386899107,2cyclictest16134-21cut11:50:152
1386899107,2cyclictest10837-21expr08:10:162
1386899100,9cyclictest6774-21ssh10:45:402
1386499108,1cyclictest6202-21tail09:05:261
1386499108,1cyclictest30917-21grep07:45:151
1386499108,1cyclictest29856-21grep08:50:131
1386499108,1cyclictest28903-21sort08:45:271
1386499108,1cyclictest18897-21ls08:25:181
1386499108,1cyclictest11771-21grep08:10:231
1386499107,2cyclictest29199-21sed07:40:171
1386499107,2cyclictest26824-21chrt08:40:301
1386499107,2cyclictest16594-21cut07:15:141
1386299108,1cyclictest4549-21apache_volume09:05:120
1386299108,1cyclictest4150-21chrt09:03:260
1386299108,1cyclictest24479-21smartctl07:30:160
1386299108,1cyclictest22541-21date08:35:010
1386299108,1cyclictest11822-21grep08:10:230
1386299107,2cyclictest16280-21grep08:20:180
1386299100,9cyclictest28415-21taskset07:40:080
138709998,1cyclictest23983-21awk08:35:213
138709998,1cyclictest2247-21awk09:00:123
138689997,1cyclictest20644-21cat08:30:152
138689997,1cyclictest17831-21taskset08:25:022
138649997,1cyclictest9821-21head08:05:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional