You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Dec 04, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
456734212777,46sleep30-21swapper/307:07:113
45670228453,10sleep10-21swapper/107:06:451
45691228265,9sleep20-21swapper/207:09:272
45664727759,12sleep00-21swapper/007:06:040
457221992818,6cyclictest673955-21taskset10:07:193
45722199241,19cyclictest741345-21taskset10:51:013
7414402238,2sleep20-21swapper/210:51:132
45722199232,15cyclictest828046-21sleep11:50:003
45722199232,15cyclictest682792-21turbostat10:14:593
457221992314,4cyclictest7490752sleep310:56:193
6461192222,15sleep0646381-21df09:50:130
457221992219,1cyclictest0-21swapper/308:20:133
457221992219,1cyclictest0-21swapper/307:19:593
457221992218,2cyclictest0-21swapper/311:13:233
45722199212,13cyclictest795118-21taskset11:27:013
457221992117,2cyclictest0-21swapper/312:15:003
457221992117,2cyclictest0-21swapper/310:40:163
457221992117,2cyclictest0-21swapper/309:21:503
457221992116,3cyclictest0-21swapper/311:20:253
45722199211,15cyclictest637541-21chrt09:43:473
45722199203,14cyclictest0-21swapper/312:22:453
45722199203,13cyclictest0-21swapper/311:00:313
457221992017,1cyclictest0-21swapper/308:35:003
457221992015,3cyclictest0-21swapper/309:35:183
45722199201,15cyclictest0-21swapper/309:10:003
45722199201,14cyclictest0-21swapper/311:50:303
45722199201,12cyclictest0-21swapper/311:06:473
7725182190,3sleep10-21swapper/111:12:581
7030002191,12sleep2703172-21/usr/sbin/munin10:25:272
6222112191,3sleep20-21swapper/209:32:082
5500872191,12sleep1550106-21apt-get08:35:111
5500872191,12sleep1550106-21apt-get08:35:111
45722199194,9cyclictest660438-21taskset10:00:063
45722199192,13cyclictest780301-21taskset11:18:073
45722199192,13cyclictest622797-21taskset09:33:163
45722199192,12cyclictest858735-21taskset12:09:423
45722199192,12cyclictest598932-21taskset09:15:203
457221991916,1cyclictest0-21swapper/309:57:013
457221991915,2cyclictest0-21swapper/307:37:233
457221991915,2cyclictest0-21swapper/307:35:013
457221991913,2cyclictest0-21swapper/309:45:283
45722199191,14cyclictest849289-21chrt12:01:293
45722199191,14cyclictest593714-21chrt09:13:483
45722199191,14cyclictest0-21swapper/310:31:313
45722199191,13cyclictest0-21swapper/307:40:153
45722199191,13cyclictest0-21swapper/307:20:143
457221991910,5cyclictest804576-21turbostat11:35:003
5936992181,2sleep10-21swapper/109:13:461
45722199185,6cyclictest462505-21taskset07:14:293
45722199182,12cyclictest898089-21taskset12:33:323
457221991815,1cyclictest0-21swapper/309:00:153
45722199181,4cyclictest695211-21taskset10:20:273
457221991814,2cyclictest0-21swapper/308:50:003
45722199181,14cyclictest719809-21chrt10:38:383
45722199181,14cyclictest0-21swapper/312:25:063
45722199181,14cyclictest0-21swapper/311:55:123
45722199181,14cyclictest0-21swapper/309:50:213
45722199181,13cyclictest899380-21chrt12:35:123
45722199181,13cyclictest688033-21chrt10:15:333
45722199181,13cyclictest0-21swapper/311:40:083
45722199181,13cyclictest0-21swapper/311:35:083
9038222171,3sleep00-21swapper/012:35:260
8907902171,4sleep0586855-21diskmemload12:29:410
7946862171,3sleep20-21swapper/211:26:172
7643182171,4sleep0586855-21diskmemload11:06:180
7575552171,3sleep0586855-21diskmemload11:02:550
6983612171,3sleep00-21swapper/010:25:090
6803342171,10sleep1680586-21users10:10:281
457221991713,2cyclictest0-21swapper/308:10:163
45722199171,13cyclictest0-21swapper/310:25:203
45722199171,13cyclictest0-21swapper/309:27:203
45722199171,12cyclictest555132-21turbostat08:40:013
45722199171,12cyclictest555132-21turbostat08:40:003
45722199171,12cyclictest0-21swapper/312:15:283
45722199171,11cyclictest529067-21chrt08:15:133
45722199170,13cyclictest0-21swapper/310:45:113
8402352162,7sleep2840284-21ntpq11:55:192
8245972161,8sleep1824698-21ntpq11:45:201
7934092161,4sleep1793512-21ntpq11:25:191
7896952161,11sleep0789872-21/usr/sbin/munin11:25:140
7430872161,3sleep10-21swapper/110:54:301
7280572161,4sleep0586855-21diskmemload10:44:450
7031952161,10sleep1703205-21ssh10:25:301
6955652160,3sleep20-21swapper/210:20:312
6370872161,3sleep10-21swapper/109:42:581
6366992161,2sleep20-21swapper/209:42:112
45722199161,9cyclictest512888-21taskset08:00:013
45722199161,3cyclictest0-21swapper/309:00:003
457221991612,2cyclictest0-21swapper/308:53:263
45722199161,11cyclictest523547-21turbostat08:10:003
45722199161,11cyclictest480469-21turbostat07:30:003
45722199161,10cyclictest507870-21taskset07:55:123
45722199160,10cyclictest501033-21taskset07:45:223
8601592151,9sleep0861030-21grep12:10:180
8197562151,8sleep10-21swapper/111:44:131
7345302151,9sleep0586855-21diskmemload10:47:340
7133612151,9sleep2713377-21/usr/sbin/munin10:35:132
7098312151,3sleep10-21swapper/110:30:201
6958132150,9sleep0586855-21diskmemload10:20:560
6522392151,3sleep2586855-21diskmemload09:53:372
6022712151,8sleep1602467-21cstates09:20:121
6015652151,9sleep0601570-21ssh09:19:190
4698862159,2sleep10-21swapper/107:20:001
457221991513,1cyclictest0-21swapper/308:40:203
457221991512,1cyclictest0-21swapper/308:25:163
45722199150,13cyclictest0-21swapper/307:50:023
9059162141,8sleep1586855-21diskmemload12:38:281
8919842141,8sleep1893295-21cut12:30:191
8564512148,1sleep20-21swapper/212:05:292
8491702140,8sleep0586855-21diskmemload12:01:200
8350322140,3sleep0586855-21diskmemload11:53:420
6310842140,3sleep0586855-21diskmemload09:40:030
6165752143,5sleep0616599-21apt-get09:30:130
6002222141,8sleep2586855-21diskmemload09:16:362
5647242141,9sleep1564782-21cat08:45:211
4908872141,8sleep1490888-40gst-plugin-scan07:37:231
8754862131,8sleep1875498-21ssh12:20:131
8731382131,3sleep1873149-21ssh12:16:431
8725052131,6sleep0586855-21diskmemload12:15:420
8650652131,5sleep20-21swapper/212:11:322
8351512131,6sleep1586855-21diskmemload11:53:541
8255362131,3sleep2825545-21ssh11:45:412
8169332131,7sleep0816985-21ntpq11:40:230
8113312131,5sleep10-21swapper/111:38:101
7978432131,7sleep2799362-21kthreadcore11:30:192
7800662131,7sleep0586855-21diskmemload11:17:450
7490142131,8sleep0749024-21ssh10:56:120
6882062131,6sleep10-21swapper/110:15:521
6756722131,5sleep00-21swapper/010:10:070
6740822131,6sleep2586855-21diskmemload10:07:352
6664802131,6sleep0586855-21diskmemload10:02:520
6661812131,6sleep1586855-21diskmemload10:02:181
6450922131,3sleep2645102-21ssh09:48:512
6290802131,6sleep2586855-21diskmemload09:36:362
5675242131,7sleep1569311-21kthreadcore08:50:181
8905202121,7sleep2890529-21ssh12:29:152
8804792121,7sleep2880489-21rm12:20:422
8417672121,5sleep0586855-21diskmemload11:57:020
8167202121,4sleep2816872-21ntp_192.168.11411:40:202
8020982121,5sleep0586855-21diskmemload11:30:490
7817042121,5sleep1781730-21ssh11:20:151
7421112121,5sleep0586855-21diskmemload10:52:400
7351922121,2sleep1735202-21ssh10:48:471
7202512121,7sleep0720256-21ssh10:39:240
6763052121,7sleep2676820-21kthreadcore10:10:172
6661472121,7sleep2666156-21sh10:02:152
6073302121,5sleep20-21swapper/209:21:062
5647832121,7sleep2564870-21/usr/sbin/munin08:45:222
5278372121,5sleep0527891-21ntpq08:10:210
4917072122,6sleep1491830-21tune2fs07:40:161
4657422121,6sleep0467431-21kthreadcore07:15:190
9068262111,3sleep20-21swapper/212:39:572
8955612112,4sleep2895619-21ntpq12:30:242
8883982110,1sleep10-21swapper/112:25:371
8825792111,6sleep0882590-21ssh12:24:090
8727932111,3sleep20-21swapper/212:16:082
8664202111,3sleep10-21swapper/112:13:511
8568512111,3sleep00-21swapper/012:06:120
8510682111,3sleep10-21swapper/112:04:451
8411252111,4sleep1586855-21diskmemload11:55:461
8342572111,6sleep2834266-21ssh11:52:192
8261812114,4sleep0457210-21cyclictest11:46:470
8111442111,3sleep20-21swapper/211:37:512
8109552111,3sleep00-21swapper/011:37:330
8019902111,3sleep10-21swapper/111:30:381
7877542111,4sleep0586855-21diskmemload11:22:430
7872722111,3sleep20-21swapper/211:21:482
7801992111,3sleep10-21swapper/111:17:581
7796452111,4sleep2586855-21diskmemload11:16:592
7643612111,3sleep10-21swapper/111:06:231
7643422111,3sleep20-21swapper/211:06:202
7568812111,3sleep10-21swapper/111:01:391
7567032111,4sleep20-21swapper/211:01:192
7499992111,4sleep2586855-21diskmemload10:57:582
7487082111,3sleep1586855-21diskmemload10:55:411
7341862111,2sleep20-21swapper/210:46:562
7187482111,3sleep10-21swapper/110:36:371
7112102111,3sleep20-21swapper/210:31:422
6884302111,3sleep20-21swapper/210:16:172
6738622114,3sleep0151ktimers/010:07:090
6734962111,4sleep1586855-21diskmemload10:06:261
6583802111,4sleep1586855-21diskmemload09:56:121
6524482111,4sleep1586855-21diskmemload09:54:161
6439472111,3sleep00-21swapper/009:46:370
6437462111,3sleep10-21swapper/109:46:141
6306592111,6sleep1630670-21ssh09:39:481
6070832111,4sleep0586855-21diskmemload09:20:350
5912772111,5sleep0591332-21ntpq09:10:210
5770612111,6sleep1577545-21kthreadcore09:00:161
5702892111,2sleep20-21swapper/208:50:282
5612622111,6sleep0562260-21kthreadcore08:45:170
5594122111,5sleep2559458-21ntpq08:40:212
5382982111,5sleep2538344-21ntpq08:20:212
5225312111,5sleep1522581-21ntpq08:05:221
4955212111,5sleep0495581-21ntpq07:40:230
4909362110,5sleep0490880-31find07:37:350
4742632111,2sleep00-21swapper/007:20:230
8964122101,4sleep0586855-21diskmemload12:30:500
8482762101,3sleep20-21swapper/212:00:242
7714962101,4sleep0586855-21diskmemload11:11:080
7714022101,3sleep2586855-21diskmemload11:10:532
7266252101,3sleep2586855-21diskmemload10:42:122
7248732101,3sleep1586855-21diskmemload10:40:171
7112292101,3sleep0586855-21diskmemload10:31:430
6971242101,3sleep1586855-21diskmemload10:23:231
6583842101,2sleep20-21swapper/209:56:132
6531462101,3sleep0586855-21diskmemload09:55:060
6292532101,3sleep0586855-21diskmemload09:36:550
6221232101,2sleep10-21swapper/109:31:581
6149032101,2sleep10-21swapper/109:27:051
6144672101,3sleep20-21swapper/209:25:342
6099692101,2sleep00-21swapper/009:25:160
6011552100,2sleep10-21swapper/109:18:281
5913982101,4sleep2591465-21ntp_states09:10:242
5805772101,4sleep2580625-21ntpq09:00:222
5659362101,5sleep0565951-21/usr/sbin/munin08:50:130
5503772101,4sleep0550586-21hddtemp_smartct08:35:150
5503772101,4sleep0550586-21hddtemp_smartct08:35:140
5458792101,4sleep0547589-21grep08:30:170
5395652101,4sleep0539826-21df_inode08:25:120
5330182101,4sleep2533066-21ntpq08:15:212
5240832101,4sleep1524207-21perl08:10:141
5185722101,4sleep2518779-21diskstats08:05:152
5169722101,4sleep0517024-21ld08:00:190
5075952101,4sleep1507614-21apt-get07:55:121
5011712101,4sleep1501388-21users07:45:251
4797712101,5sleep2479944-21sort07:25:262
4754352101,3sleep0475454-21apt-get07:25:130
4618222100,2sleep20-21swapper/207:10:242
856048290,2sleep10-21swapper/112:05:241
688197291,2sleep00-21swapper/010:15:510
571477290,6sleep20-21swapper/208:55:162
571203291,3sleep00-21swapper/008:55:150
550662291,5sleep20-21swapper/208:35:162
550662291,5sleep20-21swapper/208:35:162
534317291,3sleep1534336-21apt-get08:20:121
534034291,4sleep1534149-21wc08:20:011
512153290,1sleep00-21swapper/007:55:270
500819291,3sleep0500872-21ntp_states07:45:210
490057291,4sleep2490111-21ntpq07:35:202
470171290,3sleep2470190-21apt-get07:20:132
572130281,3sleep1573820-21tr08:55:191
548670281,3sleep2548811-21ntpq08:30:192
501590282,4sleep20-21swapper/207:47:542
476105281,3sleep1477769-21kthreadcore07:25:171
586716271,3sleep0586717-21sshd09:08:240
582893271,2sleep20-21swapper/209:05:182
581812270,2sleep10-21swapper/109:05:131
518479270,1sleep00-21swapper/008:05:050
508609272,2sleep2509659-21kthreadcore07:55:162
502260270,3sleep1501771-21apt-get07:50:021
485747271,2sleep2567-21dbus-daemon07:35:002
474016270,2sleep1474157-21ntpq07:20:201
560429260,1sleep00-21swapper/008:44:560
549613261,3sleep1331ktimers/108:32:181
543438260,2sleep20-21swapper/208:25:192
542896261,2sleep10-21swapper/108:25:171
528784261,2sleep20-21swapper/208:15:002
503574261,2sleep00-21swapper/007:50:170
496187260,4sleep20-21swapper/207:41:252
485435260,4sleep10-21swapper/107:31:061
469819260,4sleep20-21swapper/207:19:132
462079260,3sleep10-21swapper/107:10:261
560308250,2sleep10-21swapper/108:43:241
534613250,2sleep00-21swapper/008:20:150
533347250,2sleep00-21swapper/008:15:260
506472250,1sleep20-21swapper/207:50:232
484813250,2sleep00-21swapper/007:30:210
576840240,1sleep00-21swapper/009:00:150
513104240,1sleep10-21swapper/108:00:121
512618240,1sleep20-21swapper/208:00:012
457806240,1sleep00-21swapper/007:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional