You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 23:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 14, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
180077622240,2sleep10-21swapper/111:44:211
1438236212667,55sleep30-21swapper/307:06:093
16377392890,2sleep20-21swapper/209:55:172
143823928554,9sleep10-21swapper/107:06:121
143833328352,10sleep20-21swapper/207:07:252
143831627952,19sleep00-21swapper/007:07:110
1438657993834,2cyclictest0-21swapper/312:40:003
15675362350,2sleep30-21swapper/309:10:013
1438657993229,1cyclictest0-21swapper/309:45:263
1438657993215,14cyclictest541ktimers/309:55:353
1438657993026,2cyclictest0-21swapper/311:00:173
1438657993026,2cyclictest0-21swapper/310:50:003
16196912281,11sleep21567537-21diskmemload09:44:272
1438657992824,2cyclictest0-21swapper/310:30:243
1438657992822,3cyclictest66550irq/131-enp1s0-TxRx-310:22:573
1438657992817,8cyclictest0-21swapper/309:42:363
1438657992621,3cyclictest0-21swapper/312:35:013
1438657992621,2cyclictest0-21swapper/307:20:123
1438657992615,8cyclictest0-21swapper/310:44:213
1438657992613,2cyclictest541ktimers/310:05:143
1438657992523,1cyclictest0-21swapper/309:38:183
1438657992522,1cyclictest0-21swapper/307:38:243
143865799252,17cyclictest1725600-21sleep10:55:003
1438657992521,2cyclictest0-21swapper/311:34:493
1438657992519,4cyclictest15950irq/132-ahci[0000:00:17.0]12:15:133
1438657992513,2cyclictest541ktimers/312:25:083
1438657992510,2cyclictest0-21swapper/311:17:163
143865799250,19cyclictest1696650-21chrt10:35:153
1438657992422,1cyclictest0-21swapper/311:38:243
1438657992421,1cyclictest0-21swapper/311:14:223
1438657992421,1cyclictest0-21swapper/309:27:223
1438657992414,7cyclictest0-21swapper/311:06:023
1438657992413,8cyclictest0-21swapper/312:12:503
1438657992410,2cyclictest0-21swapper/309:23:013
143865799239,2cyclictest0-21swapper/310:12:013
143865799239,2cyclictest0-21swapper/309:53:183
1438657992320,1cyclictest0-21swapper/312:20:153
1438657992319,2cyclictest0-21swapper/311:23:533
1438657992319,2cyclictest0-21swapper/310:55:283
1438657992319,2cyclictest0-21swapper/307:50:013
1438657992317,3cyclictest66550irq/131-enp1s0-TxRx-312:06:513
1438657992310,4cyclictest541ktimers/312:00:143
18778002221,9sleep11567537-21diskmemload12:32:551
15453542221,9sleep21545402-21ntpq08:45:192
143865799229,4cyclictest541ktimers/309:32:133
143865799227,11cyclictest1772652-21taskset11:25:173
143865799222,13cyclictest1805990-21taskset11:45:263
1438657992218,2cyclictest0-21swapper/311:56:093
1438657992218,2cyclictest0-21swapper/311:52:533
1438657992218,2cyclictest0-21swapper/310:15:163
1438657992218,2cyclictest0-21swapper/310:00:243
1438657992218,2cyclictest0-21swapper/309:11:293
1438657992210,7cyclictest1545676-21taskset08:45:243
1438657992118,1cyclictest0-21swapper/307:40:133
1438657992117,2cyclictest0-21swapper/310:25:143
17698642201,7sleep00-21swapper/011:23:080
17026722201,6sleep00-21swapper/010:38:090
16966162201,8sleep21567537-21diskmemload10:35:152
143865799209,6cyclictest1494063-21taskset08:00:143
143865799206,10cyclictest1540776-21taskset08:40:343
1438657992018,1cyclictest0-21swapper/308:05:453
1438657992017,2cyclictest0-21swapper/308:50:173
1438657992017,1cyclictest0-21swapper/309:15:193
1438657992017,1cyclictest0-21swapper/308:30:193
1438657992016,2cyclictest0-21swapper/311:40:283
1438657992016,2cyclictest0-21swapper/307:53:073
1438657992013,3cyclictest15150992sleep308:20:113
1438657991916,2cyclictest0-21swapper/309:00:143
1438657991916,1cyclictest0-21swapper/308:25:243
1438657991916,1cyclictest0-21swapper/308:10:163
1438657991915,2cyclictest0-21swapper/307:25:173
143865799191,13cyclictest0-21swapper/308:35:143
18233892181,10sleep01567537-21diskmemload11:58:280
16879732181,4sleep20-21swapper/210:28:412
16877662181,3sleep00-21swapper/010:28:190
16788702181,2sleep20-21swapper/210:20:402
16565502180,4sleep21656560-21ssh10:06:322
16503802181,11sleep21567537-21diskmemload10:04:162
14721872180,11sleep11472172-31find07:38:261
1438657991816,1cyclictest0-21swapper/307:33:033
1438657991815,2cyclictest0-21swapper/308:58:353
1438657991815,2cyclictest0-21swapper/307:20:003
1438657991815,2cyclictest0-21swapper/307:13:273
1438657991815,1cyclictest0-21swapper/308:17:253
18379692171,5sleep11567537-21diskmemload12:05:581
18012292171,9sleep11567537-21diskmemload11:45:011
17767052171,8sleep01776716-21sh11:27:040
16803352171,3sleep10-21swapper/110:23:281
16577552171,3sleep10-21swapper/110:08:401
16334572171,10sleep21567537-21diskmemload09:51:452
15873372171,3sleep00-21swapper/009:20:490
143865799171,4cyclictest1488755-21taskset07:55:023
17679302161,3sleep20-21swapper/211:20:212
17558422161,4sleep01567537-21diskmemload11:14:340
16498212161,3sleep00-21swapper/010:03:140
16334862161,10sleep11567537-21diskmemload09:51:491
15873572161,4sleep21567537-21diskmemload09:20:502
18870992151,9sleep11887109-21ssh12:39:421
18701082151,8sleep11567537-21diskmemload12:27:591
18646162151,6sleep00-21swapper/012:25:180
18468502151,7sleep20-21swapper/212:13:012
17921912151,3sleep10-21swapper/111:37:551
17855152151,10sleep11785527-21ssh11:34:521
17690282151,8sleep11567537-21diskmemload11:21:171
17568732151,9sleep21757007-21ssh11:15:152
17406362151,7sleep11740668-21munin-run11:05:011
16944792151,10sleep01694493-21sh10:31:450
16881252151,7sleep11567537-21diskmemload10:29:001
14721512151,8sleep01472172-31find07:38:240
14503872150,1sleep00-21swapper/007:15:240
18862232140,1sleep00-21swapper/012:38:080
18246992141,8sleep21824713-21ssh12:00:062
18081482141,8sleep21808153-21ssh11:48:552
17107252140,2sleep01567537-21diskmemload10:44:280
17085992141,2sleep10-21swapper/110:40:281
16560392141,8sleep01656045-21ssh10:05:300
16504482141,8sleep11567537-21diskmemload10:04:241
16265282141,9sleep11626543-21ssh09:48:031
16095582140,2sleep00-21swapper/009:35:260
15754252141,9sleep11575489-21ls09:15:121
15457082141,8sleep01545789-21grep08:45:250
14927092141,3sleep01492757-21ntpq07:55:190
14431602141,2sleep20-21swapper/207:10:242
18766022131,6sleep00-21swapper/012:30:500
18559582131,7sleep21855968-21ssh12:19:532
18401882131,5sleep00-21swapper/012:09:540
18164082131,8sleep01816417-21sh11:54:500
17927112131,5sleep00-21swapper/011:38:500
17844152131,7sleep01567537-21diskmemload11:32:450
17099922131,3sleep21709998-21ssh10:43:012
16723772131,1sleep20-21swapper/210:17:472
15940342131,8sleep11594102-21/usr/sbin/munin09:25:221
15818272131,6sleep01567537-21diskmemload09:19:480
14997132131,8sleep21499309-21/usr/sbin/munin08:05:162
14393682131,8sleep01439772-21cut07:10:140
18429072121,7sleep01844558-21latency12:10:170
18315172121,4sleep01567537-21diskmemload12:03:440
18307902121,5sleep11567537-21diskmemload12:02:291
17617552121,5sleep01567537-21diskmemload11:16:420
17547272120,2sleep10-21swapper/111:12:271
17462652121,7sleep21746272-21sh11:05:532
17339302120,1sleep00-21swapper/011:00:190
17032322121,4sleep11567537-21diskmemload10:39:101
16261762121,5sleep20-21swapper/209:47:242
16041202121,7sleep21567537-21diskmemload09:33:572
15295822121,5sleep11529620-21ntpq08:30:191
14945902121,7sleep21495189-21kthreadcore08:00:182
14879182121,5sleep11488135-21users07:50:271
14731542121,7sleep21473397-21kthreadcore07:40:142
14557372125,3sleep11455840-21timerandwakeup07:20:251
18616032111,4sleep21567537-21diskmemload12:21:152
18486652112,3sleep11848685-21apt-get12:15:121
18465202111,3sleep10-21swapper/112:12:271
18331442111,4sleep21833364-21hddtemp_smartct12:05:142
18221862111,4sleep11567537-21diskmemload11:56:191
18156222111,3sleep11567537-21diskmemload11:53:311
18155962111,4sleep21567537-21diskmemload11:53:292
18014402111,3sleep00-21swapper/011:45:050
18006292110,5sleep21567537-21diskmemload11:44:072
17982332111,5sleep01798489-21ssh11:40:270
17769632111,4sleep11567537-21diskmemload11:27:331
17622302111,3sleep10-21swapper/111:17:311
17558612111,4sleep21567537-21diskmemload11:14:362
17462532111,3sleep00-21swapper/011:05:510
17405652111,5sleep21740575-21ssh11:04:542
17318452111,3sleep20-21swapper/210:57:312
17235792111,4sleep21567537-21diskmemload10:51:032
17228112111,4sleep11567537-21diskmemload10:50:261
17162332111,4sleep11567537-21diskmemload10:45:521
16951872111,3sleep10-21swapper/110:33:101
16799072111,3sleep00-21swapper/010:22:350
16658682111,4sleep21567537-21diskmemload10:14:552
16641712111,2sleep10-21swapper/110:11:211
16407252111,3sleep10-21swapper/109:56:091
16395822112,4sleep01639638-21ntpq09:55:200
16270702111,4sleep01567537-21diskmemload09:49:020
16193632111,4sleep01567537-21diskmemload09:43:540
16112612111,3sleep10-21swapper/109:38:091
16026292112,6sleep11602634-21ssh09:31:161
15807412111,3sleep20-21swapper/209:17:212
15519562110,5sleep10-21swapper/108:55:121
15472292111,6sleep21548392-21kthreadcore08:50:162
15401582111,5sleep21540207-21ntpq08:40:222
15364682110,7sleep01536681-21if_enp1s008:40:170
15294442110,7sleep01525622-21/usr/sbin/munin08:30:190
15098272111,7sleep11510010-21expr08:15:121
18868672101,3sleep20-21swapper/212:39:192
18784012101,4sleep21567537-21diskmemload12:33:482
18626982101,3sleep00-21swapper/012:23:120
18541722101,3sleep01567537-21diskmemload12:16:400
18214802101,3sleep21567537-21diskmemload11:55:282
17909152101,2sleep20-21swapper/211:35:442
17476582101,3sleep11567537-21diskmemload11:08:251
17308142100,2sleep10-21swapper/110:55:381
17227122101,4sleep01722808-21ps10:50:250
17170222101,3sleep00-21swapper/010:47:470
17161782101,3sleep21567537-21diskmemload10:45:452
16939432100,4sleep21567537-21diskmemload10:30:472
16715922101,3sleep11567537-21diskmemload10:16:211
16658462101,3sleep00-21swapper/010:14:530
16333522100,1sleep00-21swapper/009:51:360
16184102101,3sleep10-21swapper/109:42:111
16097462101,2sleep20-21swapper/209:35:332
16024002101,3sleep01567537-21diskmemload09:30:430
15891902101,4sleep11567537-21diskmemload09:24:391
15739362101,3sleep01567537-21diskmemload09:12:520
15572932101,4sleep11557554-21df_inode09:00:141
15361432101,3sleep10-21swapper/108:40:141
14608432101,5sleep11460959-21gzip07:25:231
14558612101,4sleep21456057-21users07:20:262
1869733290,3sleep21567537-21diskmemload12:27:152
1862016291,2sleep10-21swapper/112:21:531
1771869294,2sleep20-21swapper/211:25:172
1730765291,2sleep00-21swapper/010:55:330
1671583291,2sleep00-21swapper/010:16:200
1595908290,1sleep20-21swapper/209:28:282
1595580291,2sleep00-21swapper/009:27:540
1573019291,2sleep10-21swapper/109:11:001
1563120291,4sleep21564747-21kthreadcore09:05:152
1526224291,4sleep21527770-21cut08:30:162
1515122291,4sleep01515374-21df08:20:130
1510313291,4sleep01510419-21grep08:15:150
1494322291,4sleep01494392-21grep08:00:170
1477981290,3sleep21478000-21apt-get07:45:122
1462560291,4sleep21463786-21cut07:30:162
1458819291,4sleep01460552-21cat07:25:180
1451995291,4sleep01453475-21cut07:20:160
1572875280,1sleep20-21swapper/209:10:442
1555830280,4sleep01555969-21/usr/sbin/munin08:55:190
1552778281,3sleep21554028-21kthreadcore08:55:172
1531420281,3sleep20-21swapper/208:35:172
1509545281,3sleep11509570-21perf08:15:001
1508548281,3sleep21508600-21/usr/sbin/munin08:10:202
1450150282,3sleep1331ktimers/107:15:211
1783148270,4sleep20-21swapper/211:30:322
1562010270,1sleep20-21swapper/209:01:542
1534950270,1sleep00-21swapper/008:35:230
1522307271,2sleep00-21swapper/008:25:210
1520354270,2sleep20-21swapper/208:25:162
1508044270,2sleep01508492-21strings08:10:190
1504026271,3sleep1802-21snmpd08:06:551
1499022271,3sleep1246-21systemd-journal08:04:421
1493604270,1sleep20-21swapper/207:58:112
1485416270,3sleep01487249-21kthreadcore07:50:190
1483527271,2sleep21483546-21apt-get07:50:122
1461988270,1sleep00-21swapper/007:30:100
1456740270,2sleep20-21swapper/207:25:122
1446092270,1sleep20-21swapper/207:15:102
1443855271,2sleep11443899-21systemd-run07:15:001
1562517260,2sleep10-21swapper/109:05:101
1557838261,2sleep00-21swapper/009:00:170
1546305260,1sleep10-21swapper/108:48:511
1519970260,1sleep20-21swapper/208:23:352
1515375261,2sleep10-21swapper/108:20:131
1514600260,1sleep20-21swapper/208:17:192
1499332261,2sleep00-21swapper/008:05:130
1489348260,2sleep10-21swapper/107:55:161
1482662260,1sleep10-21swapper/107:45:501
1482306261,2sleep00-21swapper/007:45:250
1477718260,4sleep10-21swapper/107:44:581
1477480260,1sleep00-21swapper/007:42:010
1472053260,1sleep20-21swapper/207:37:432
1466741260,1sleep10-21swapper/107:32:041
1550619250,1sleep10-21swapper/108:50:201
1550216251,1sleep00-21swapper/008:50:190
1562299240,1sleep00-21swapper/009:05:010
1530612240,1sleep10-21swapper/108:35:001
1520644240,1sleep10-21swapper/108:25:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional